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PREFACE 


This  is  Volume  I  of  the  Cost-Effectiveness  Specification  for  Computer- 
Based  Training  Systems.  It  is  part  of  a  three-volume  set  which  corresponds 
to  the  three  phases  of  a  training  system's  life  cycle  (Volume  I  - 
Development;  Volume  II  -  Procurement;  Volume  III  -  Operation  and  Mainten¬ 
ance).  An  Executive  Summary  document  is  included  which  provides  the 
reader  with  general  guidance  and  instructions  on  how  to  go  through  and 
respond  to  the  various  parts  of  the  specification.  This  Cost-Ef fectivenes 
Specification  was  developed  for  DARPA  as  part  of  Contract  #MDA903-76-C-0210. 
Dr.  Harold  F.  O'Neil  was  the  Technical  Monitor. 
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Part  A 

SYSTEM  DESCRIPTION 
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INSTRUCTIONS 


The  purpose  of  Part  A  is  to  describe  some  of  the  characteristics  of 
the  computer-based  training  system  and  to  identify  the  individuals  who 
are  using  this  specification. 


Print  the  name  or  descriptive  title  of  the  computer-based  training 
•ystem. 


© 


Enter  the  year  which  represents  the  system's  current  life  cycle 
status.  Indicate  which  year  of  the  Development  Phase  you  are  in  by 
entering  the  date  in  the  appropriate  space. 


CD 


Check  the  type  of  hardware  configuration  supporting  the  system. 


© 


Enter  the  maximum  number  of  time-sharing  students  per  system. 


© 


List  the  courses  to  be  taught  or  supported  by  the  computer-based 
training  system. 


© 


Estimate  the  number  of  students  who  are  to  be  served  by  the  system 
in  each  course,  and  in  all  courses  per  year. 


2)  Print  your  name,  job  title,  and  organization. 


SYSTEM  DESCRIPTION 


© 

© 


Computer-Based  Training  System, 


System  Life  Cycle 

Phase  I  -  Development 


(3)  Configuration 

□  Stand-Alone 


(4)  Time-Sharing  Capacity 


1st  Year  2nd  Year  3rd  Year  4th  Year  5th  Year  6th  Year 

19  19  19  _  19  _  19 _ 19 _ 


□ 


Remote 


(5)  Courses  Supported  by  System  (Specify) 


□ 


Combination  Stand-Alone  and  Remote 


No.  of  Students 
Per  Year 


Pc.rt  B 

GENERAL  COSTING  ASSUMPTIONS  AND  DEFINITIONS 


1L 
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Part  B 


GENERAL  COSTING  ASSUMPTIONS  AND  DEFINITIONS 
DEVELOPMENT  PHASE 

Activities  related  to  the  design  and  production  of  a  tested,  prototype 
computer-based  training  system.  Development  costs  are  all  the  costs 
resulting  from  applied  research,  engineering  design,  analysis,  development, 
test,  evaluation  and  managing  of  development  efforts  related  to  a  specific 
computer-based  training  system.  Included  are  all  costs  (in-house  and 
contractor),  irrespective  of  how  funded,  which  are  necessary  to  develop  a 
tested  prototype  of  a  specific  computer-based  training  system.  These  costs 
include  the  following  components: 

•  Equipment.  The  costs  incurred  during  study,  analysis,  design, 
development,  evaluation,  testing,  and  redesign  of  the  equipment  for  the 
tested  prototype  system.  It  includes  the  design  effort  of  preparing  speci¬ 
fications,  engineering  drawings,  parts  lists,  wiring  diagrams,  test  planning 
and  scheduling,  analysis  of  test  results,  data  reduction,  report  preparation 
and  establishment  of  reliability,  maintainability  and  quality  assurance 
control  requirements.  It  also  includes  the  cost  of  raw  and  semifabricated 
material  plus  purchased  parts  consumed  in  the  performance  of  component 
engineering  efforts.  It  also  contains  the  cost  of  fabrication,  processing, 
subassembly,  final  assembly,  reworking,  modification,  and  installation  of 
parts  and  equipment  in  the  prototype  system.  Included  also  are  engineering 
test  equipment  such  as  oscilloscopes  required  to  accomplish  this  function 
within  the  Development  Phase. 

•  Fdcil ities.  The  costs  of  any  new  building,  conversion  or  expansion 
of  facilities  or  sites,  and  the  acquisition  of  real  estate  for  development 
and  testing  of  the  computer-based  training  system.  Some  of  these  costs  can 
be  determined  by  use  of  military  guides  which  provide  standards  for  such 
facilities. 

•  Software.  The  costs  of  programming  the  computer  to  support  the 
training  system. 

t  Instructional  System  Development  (ISP)  Activities.  The  costs  of 
performing  all  activities  in  the  first  three  phases  of  the  ISD  process 
except  preparation  of  instructional  materials  and  tests. 

•  Instructional  Methods/Materials.  The  costs  of  preparing  all  the 
training  and  testing  materials  to  be  used  in  the  prototype  system. 

•  System  Manaqement/Test.  The  costs  associated  with  managing  the 
overall  development  effort  and  operational  testing. 
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•  Other  Direct  Costs.  Any  direct  development  costs  not  included  in 
the  previous  components. 

COMPUTER-BASED  TRAINING  SYSTEM 

The  application  focus  of  this  costing  specification  is  formal ,  school- 
based  training.  The  specification  is  oriented  toward  training  which  is 
administered,  aided,  or  managed  by  computer  (hence,  computer-based) .  The 
system  component  costs  are  distributed  across  the  training  system's  life 
cycle  which  consists  of  three  phases:  Development,  Procurement,  Operation 
and  Maintenance. 

CONSTANT  DOLLARS 

In  order  to  make  valid  comparisons  between  alternatives,  the  cost  for 
each  alternative  must  be  stated  in  the  same  terms.  In  Part  D  of  the 
specification  all  costs  are  to  be  adjusted  so  that  they  are  expressed  in 
constant  year  dollars.  Constant  year  dollars  express  all  costs  expended  in 
various  years  in  terms  of  the  general  purchasing  power  of  the  dollar  for  a 
given  base  year.  An  estimate  is  said  to  be  in  constant  dollars  if  costs  for 
all  work,  both  prior,  current,  and  future,  are  adjusted  so  that  they  reflect 
the  level  of  prices  of  the  base  year.  For  purposes  of  this  specification , 
the  base  year  is  the  calendar  year  in  which  the  costing  analysis  is  performed. 

If  cost  data  or  estimates  are  available  in  other  than  constant  year 
dollars,  constant  year  dollars  are  arrived  at  by  applying  the  appropriate 
adjustment  factors.  Current  dollars  reflect  purchasing  power  current  to 
the  year  in  which  they  are  expended.  Prior  costs  stated  in  current  dollars 
are  the  actual  amounts  paid  out  in  these  years.  Future  costs  stated  in 
current  dollars  are  the  projected  actual  amounts  which  will  be  paid.  Care 
should  be  exercised  to  preclude  the  mixing  of  current  dollars  with  constant 
dollars  in  a  single  display  of  costs.  Any  cost  figures  provided  in  current 
dollars  are  to  be  clearly  identified  as  such. 

DISCOUNTED  RATES 

Discounting  is  a  technique  for  converting  various  cash  flows  occurring 
over  time  to  equivalent  amounts  at  a  common  point  in  time,  considering  the 
time  value  of  money  to  facilitate  a  valid  comparison.  The  appropriate 
interest  rate  is  used  to  discount  or  calculate  future  costs  and  benefits  so 
as  to  arrive  at  their  present  values.  Each  year's  expected  yearly  cost  is 
multiplied  by  its  discount  factor  and  then  summed  over  all  years.  (The 
current  discount  rate  specified  by  OSD  is  10  percent.) 

INDIRECT  COSTS 

An  overhead  or  indirect  cost  factor  is  attached  or  "burdened"  atop 
direct  costs  by  each  contractor  to  account  for  general  support  and  admin¬ 
istrative  expenses.  At  each  Government  or  sponsor  site  used  during  the 
life  cycle  of  the  training  system,  an  indirect  cost  factor  needs  to  be 
established.  If  no  specific  indirect  factor,  per  site,  can  be  determined, 
a  "standard"  service  or  DoD  rate  will  be  used. 


INHERITED  ASSETS 


The  use  of  assets  already  available  (inherited  assets)  requires  care¬ 
ful  evaluation  when  preparing  a  cost  estimate.  The  fact  that  a  given 
system  component  is  already  available  does  not  automatically  equate  to 
zero  cost  to  the  computer-based  training  system.  Each  inherited  asset 
must  be  evaluated  on  its  own  merit  and  in  terms  of  whether  its  use  in 
connection  with  the  system  being  costed  will  cause  some  future  expense. 

If  so,  that  expense  must  be  included  in  some  cost  element  in  the  system's 
life  cycle.  If  there  will  be  no  future  expense,  the  cost  of  the  item 
will  be  included  in  the  total  life  cycle  cost  but  will  be  highlighted  as 
a  sunk  cost.  Such  existing  assets  will  be  included  at  their  fair  market 
value  (as  measured  by  market  price,  scrap  value,  or  alternative  use)  and 
the  basis  for  arriving  at  the  estimate  will  be  documented. 

INSTRUCTION 

An  organized,  open  information-exchange  process  in  which  the  student 
and  instructional  agent  (human,  programmed  text,  or  intelligent  machine) 
continually  interact.  The  purpose  of  the  interaction  is  tc  enable  the 
student  to  reach  some  criterion  of  understanding  or  skill  proficiency 
called  mastery  of  a  given  set  of  objectives.  A  computer  can  aid  this 
process  directly  by  reducing  all  or  portions  of  a  strategy  for  inter¬ 
action  to  an  explicit  algorithm  or  set  of  algorithms.  The  computer  can 
also  help  by  providing  accurate,  reliable  massive  storage  and  retrieval 
for  records  of  student  progress. 

INSTRUCTIONAL  SYSTEM  DEVELOPMENT  (ISD) 

"A  systematic  procedure  for  assuring  application  of  instructional 
technology  to  course  planning  and  development . "1  The  five  phases  of  ISD 
are  treated  in  detail  in  the  five  volumes  of: 

Air  Force — "Handbook  for  Designers  of  Instructional  Systems'^ 

Army — "Interactive  Procedures  for  Instructional  Systems  Development'^ 

Navy /Marines — "Interservice  Procedures  for  Instructional  Systems 
Deve lopment"^ 


^-''Instructional  System  Development,"  Department  of  the  Air  Force,  AFM  50-2, 
1  July  1975,  p.  1. 1. 

2AFP  50-58,  15  July  1973. 

^RADOC  PAM  350-30,  1  August  1975. 

4NAVEDTRA106-A,  August  1975. 
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There  is  some  disagreement  between  the  Services  on  how  the  Phases  are 
numbered.  In  this  specification,  we  will  adopt  the  Army's  terminology 
of:  Analyze,  Design,  Develop,  Implement  and  Control  to  represent  Phases 
I-V,  respectively. 

An  issue  to  consider  in  working  with  this  specification  is  that  the 
instructional  development  process  does  not  coincide  with  the  computer- 
based  training  system  life  cycle  for  hardware  and  software.  The  ISD 
Phases  do  not  fit  the  Development,  Procurement,  and  Operation  and 
Maintenance  Phases  of  the  system's  life  cycle  unless  it  is  a  completely 
new  course  of  instruction.  For  most  training,  sections  of  the  course 
will  be  designed,  developed,  tested,  revised,  adapted,  and  integrated 
throughout  the  entire  life  cycle  of  the  computer-based  training  system. 

(For  example,  one- third  of  Air  Force  instructional  hours  are  revised 
each  year  on  the  average.) 

One  problem  in  obtaining  cost  data  for  ISD  activities  is  that 
functionally  related  cost  data  are  generally  not  available  from  existing 
records.  That  is,  information  regarding  personnel  time  and  non-personnel 
costs  related  to  such  functions  as  job  analysis,  media  selection,  or 
preparation  of  learning  objectives,  are  generally  not  recorded.  Thus,  such 
data  must  be  collected  during  these  activities  or  projected  by  the  analyst 
who  is  complying  with  this  specification.  One  approach  would  be  to 
designate  an  individual  with  the  responsibility  of  maintaining  a  detailed 
log  of  on-going  instructional  system  development. 

For  purposes  of  this  specification,  the  costs  associated  with  all 
activities  related  to  the  first  three  ISD  Phases  (Analyse,  Design,  and 
Develop)  will  be  documented  as  part  of  the  Development  Phase  of  the  system 
life  cycle.  The  costs  associated  with  the  ISD  activities  of  Phases  IV  and 
V  (Implement  and  Control)  will  be  accounted  for  in  the  Operations  and 
Maintenance  Phase.  No  ISD  activities  will  be  shown  as  part  of  the 
Procurement  Phase — but  rather,  this  Phase  will  involve  only  the  cost  of 
purchasing  the  completed  instructional  products,  materials  and  programs. 

LIFE  CYCLE/LIFE  CYCLE  COSTS 

The  life-cycle  cost  is  the  total  cost  of  an  item  or  system  over  its 
full  life.  The  computer-based  training  system  life  cycle  is  encompassed 
in  three  phases:  Development,  Procurement,  and  Operation  and  Maintenance. 
These  three  phases  are  of  variable  lengths  depending  upon  each  specific 
system.  Although  the  three  phases  overlap,  for  purposes  of  this  specifi¬ 
cation,  they  will  be  considered  to  occur  sequentially.  The  Development 
Phase  can  take  up  to  6  years,  the  Procurement  Phase  is  shown  as  one  year, 
and  the  Operation  and  Maintenance  Phase  can  be  up  to  8  years.  Although 
Procurement  can  take  more  than  one  year,  this  specification  requires  an 
assumption  of  all  Procurement  activities  occurring  within  the  year  follow¬ 
ing  development  of  a  tested  prototype  system.  Also,  although  such 
systems  are  usually  phased  in,  it  is  assumed  in  this  specification  that 
Procurement  will  be  followed  by  Instantaneous  operation  of  all  systems 
acquired. 
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The  specific  years  in  which  each  Phase  occurs  are  to  be  noted,  the 
information  coming  either  from  the  contractor  or  service  monitor.  The 
commitment  in  the  contract  can  provide  the  years  of  the  Development  Phase. 

Life  cycles  are  different  for  various  system  components.  These  life 
times  established  by  the  Economic  Analysis  Handbook ,  2nd  edition ,  Department 
of  Defense,  are  as  follows: 


ADP  Equipment  8  years 

Buildings  25  " 

Operating  Equipment  10  " 

Utilities,  Plants  &  Utility 

Distribution  Systems  25  " 

Weapon/Support  Systems  Variable 


PAY  AND  ALLOWANCES 

The  cost  of  civilian  and  military  personnel  paid  at  annual  rates  will  be 
gross  pay  in  current  pay  tables,  plus  the  Government's  contribution  for 
civilian  retirement,  disability,  health,  life  insurance,  and,  where 
applicable,  social  security  programs.  Factors  to  weight  the  base  pay  of 
civilian  and  military  personnel  will  be  based  on  guidance  from  the  Office 
of  the  Assistant  Secretary  of  Defense  (Comptroller) ,  "Economic  Cost  of 
Military  and  Civilian  Personnel,"  and  recent  policies  of  the  Office  of 
Management  and  Budget  regarding  civilian  personnel  pay.  The  latter  will 
be  used  to  weight  the  pay  of  civilian  Government  personnel  who  are  directly 
involved  in  the  life  cycle  of  the  computer-based  training  system.  The 
following  percentages  of  base  pay  will  be  used  in  computing  the  costs  of 
civilian  personnel  services: 

Retirement  24.7% 

Health  Insurance  3.5% 

Life  Insurance  .5% 

The  military  and  civilian  pay  rates,  as  weighted,  do  not  include  special 
pay,  such  as  flying  pay  or  hazardous  duty  pay.  These  costs  must  be  added 
to  the  rates  whenever  they  are  required  by  the  job  or  location.  If  appro¬ 
priate,  pay  should  be  increased  to  cover  leave  and  other  benefits  such  as 
the  average  cost  of  sick  leave  taken  and  annual,  holiday  and  other  paid 
leave  accruals,  plus  the  average  Government  contributions  for  other  benefits. 

SUNK  COST 

A  cost  which  is  irrevocably  committed  to  a  project.  Each  cost  analysis 
will  make  explicit  any  cost  which  is  sunk  at  the  time  the  analysis  is 
prepared.  All  costs  which  reflect  irreversible  decisions  will  be  treated 
as  sunk. 
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Part  C 

COMPUTER-BASED  TRAINING  SYSTEM  ELEMENTS: 
DEFINITIONS  AND  COSTS 


Part  C 


COMPUTER-BASED  TRAINING  SYSTEM  ELEMENTS: 
DEFINITIONS  AND  COSTS 


In  this  section  you  are  to  provide  costing  information  for  each 
element  of  the  computer-based  training  system.  For  Equipment  or  Facili- 
ties,  four  items  of  information  are  required  for  accurate  costing:  the 
type  of  device  or  unit,  the  unit  cost,  the  number  of  units  or  the 
proportion  of  one  unit  allocated  to  the  system,  and  whether  or  not  the 
units  are  to  be  supplied  by  the  government  (e.g. ,  GFE)  or  by  the 
contractor.  If  you  are  a  contractor  and  the  units  are  to  be  provided  by 
the  government,  check  the  GFE  space;  otherwise,  give  the  unit  cost. 
Maintenance  costs  should  be  included  wherever  applicable.  All  costs 
should  be  presented  in  the  year  expended.  You  should  make  a  copy  of  the 
Part  C  worksheets  for  each  year  covered  in  the  Development  Phase.  You 
can  then  easily  transfer  the  yearly  costs  to  the  matrix  in  Part  D. 

Definitions  of  all  system  elements  are  provided  on  the  left-hand 
pages,  with  examples  of  how  these  items  are  to  be  costed.  For  Software, 
Instructional  Systems  Development  (ISP)  Activities,  Instructional  Methods/ 
Materials,  and  System  Management /Test,  units  are  hard  to  define  and  unit 
costs  are  difficult  to  obtain  or  are  not  applicable.  In  these  areas,  you 
are  to  provide  as  meaningful  a  description  of  the  item  and  its  character¬ 
istics  as  possible,  and  delineate  the  personnel  and  non-personnel  costs 
which  were  incurred.  If  items  were  purchased,  and  only  total  costs  are 
available,  enter  them  in  the  appropriate  column.  To  assist  you  in  iden¬ 
tifying  total  personnel  costs,  Personnel  Cost  Worksheets  accompany  this 
specification. 
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DEFINITIONS 


EQUIPMENT 

Included  In  this  category  are  all  the  components  of  equipment  related  to  a 
computer-based  training  system  (e.g.,  the  computer  and  its  associated  auxiliary 
memory  requirements,  terminals,  carrels,  auxiliary  audiovisual  devices).  Also 
included  are  local  interface  hardware,  telephone  lines,  special  lines,  satellites, 
receivers,  power  generating  equipment,  associated  test  and  checkout  equipment, 
etc.  Maintenance  costs,  derived  from  factors  such  as  mean  time  before  failure 
and  mean  time  or  cost  to  repair,  should  be  included  in  the  costs  of  every  piece 
of  equipment  maintained. 


EXAMPLE 
1.  EQUIPMENT 

1.3  Auxiliary  Audiovisual  Devices 
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Computer (s ) :  This  refers  to  the  hardware,  either  contractor  or  in-house, 
required  to  produce  the  automatic  data  processing  capability  of  the  system.  It 
includes  the  installed  machine  or  group  of  inter-connected  machines  consisting  of 
input,  storage,  computing,  control  and  output  devices  which  use  circuitry  in  the 
main  computing  element  to  automatically  perform  arithmetic  and/or  logical  opera¬ 
tions  by  means  of  internally  stored  or  externally  controlled  programed  instructions. 
This  element  includes,  for  example,  a  central  processor,  large  capacity  storage 
data  channels,  and  input /output. 

Terminal (s) :  Refers  to  the  hardware  required  to  produce  the  data  display 
portion  of  the  system.  It  includes  the  equipment  necessary  to  provide  visual  pre¬ 
sentation  of  processed  data  or  instruction  by  means  of  specifically  designed 
electronic  or  electromechanical  devices  interconnected  with  the  computing/processing 
subsystem,  such  as  flat  panel  displays,  projection  screens,  image  data  storage 
and  retrieval  equipment. 

Auxiliary  Audiovisual  Devices:  This  subcategory  consists  of  equipment  which 
presents  audiovisual  instructional  material.  Included  are  such  devices  as: 
slide  projectors,  slide/tape  units,  film  strip  and  overhead  projectors,  various 
movie  film  projectors,  videotape  recorders/players,  etc. 


1.  EQUIPMENT 
1.1  Computer(s) 


1.3  Auxiliary  Audiovisual  Devices 


(To  Part  D  p.  61) 


f 


Auxiliary  Memory:  Storage  capability  designed  to  extend  the  core  memory  of 
a  computer.  It  may  consist  of  serial  access  devices  (e.g.,  magnetic  tape)  or 
randan  access  (e.g.,  disc  or  drum)  or  possibly  extended  core  storage  (ECS)  or 
shift  registers  added  to  the  main  memory.  The  medium  is  electro-magnetic 
although  paper  tape  could  be  used. 


Local  Interfaces:  Direct  transmission  connectors  linking  various  peripheral 
devices  (disc,  tape  drivers,  CRTs,  etc.)  with  the  computer  or  in  some  cases  with 
each  other. 


Telephone  Lines:  Standard  public  commercial  voice  grade  channels  (generally 
up  to  3000  cps)  usable  in  computer-based  training  systems. 
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1.4  Auxiliary  Memory 


1.6  Telephone  Lines 


Special  Lines:  Electrical  coamuni;  at  ion  channels  designed  to  handle 
computer  data  transmission  between  two  or  more  points.  The  circuits  are  con¬ 
structed  to  transmit  baud  rates  (or  bits  per  second)  relevant  to  various  grades 
of  transmission;  e.g.,  300-3000  cps  for  voice  grade  channels  or  up  to  50,000 
baud  (or  higher)  for  high-speed,  wide-band  digital  transmission. 


Satel 1 ites :  Earth  orbiting  electronic  transmitters  designed  as  high  altitude 
means  to  cover  a  large  geographic  area  and  to  reduce  signal  interference  caused 
by  high  natural  or  man-made  structures.  Satellite  transmission  is  typically  at 
higher  frequencies  than  is  standard  for  receivers  and  therefore  requires  frequency 
converters  at  reception  points. 


Receivers :  Devices  at  the  instructional  site  designed  to  capture  the  trans¬ 
mitted  signals  to  convert  them  to  instructor/ student  usable  form  as  a  basis  for 
training  materials.  These  include,  for  example,  radios,  TV  monitors,  ground 
receiving  equipment  relevant  to  satellite  signals,  etc.,  along  with  related 
equipment  such  as  antenna,  mount,  amplifiers,  converter,  batteries  and  cable. 
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Power  Generating  Equipment:  Where  non-electrified  areas  (or  minimal  power 
areas)  are  used  as  reception  sites  electrical  generating  equipment,  or  batteries 
are  required  to  activate  receivers  (or  amplify)  transmittal  signals  over  appro¬ 
priate  receivers. 


Carrels:  Small  enclosures  for  individual  study  that  are  moveable.  If 
firmly  fixed  or  built  into  the  structure  of  the  building,  they  would  be 
categorized  under  Facilities. 
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1.10  Power  Generating  Equipment 


Description 


a. 

b. 

c. 

d. 

e. 


1.11  Carrels 

a.  _ 

b.  _ 

c.  _ 

d.  _ 


e. 


1.12  Other  Equipment 


FACILITIES 


Included  in  this  category  are  all  the  physical  facilities  required  for  housing 
the  equipment  components,  administrators,  and  users  of  the  computer-based  training 
system.  This  category  includes  classrooms,  laboratories,  large  group  instruction 
spaces,  offices,  individual  learning  spaces,  libraries  and  other  information 
resource  centers,  etc.  Units  should  be  specified  as  the  number  of  square  feet 
required  to  house  all  components  of  the  computer-based  training  system. 


EXAMPLE 
2.  FACILITIES 
2.4  OH  ices 
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Classrooms :  Places  within  buildings  in  which  training  is  administered  to 
students. 


Laboratories:  Places  providing  opportunity  for  observation,  practice,  or 

experimentation.  Included  in  this  category  are  simulated  job  settings  which  can 
be  room-size  or  as  large  as  warehouses,  depending  upon  the  particular  training 
situation. 


Large  Group  Instruction  Spaces:  Included  in  this  element  are  such  places  as 
auditoria,  study  halls,  demonstration  rooms,  etc.,  where  large  numbers  of  students 
can  be  trained. 


Offices :  Places  in  which  the  personnel  who  develop,  manage,  administer, 

and  support  the  computer-based  training  system  perform  their  functions. 
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2.  FACILITIES 
2.1  Classrooms 

Description 


b.  _ 

c.  _ 

d.  _ 

e.  _ 

2.2  Laboratories 


1 


(To  Part  D  p.  61 ) 


e. 


2.3  Large  Group  Instruction  Spaces 

i _  _ 


2.4  Offices 


Individual  Learning  Spaces:  This  category  includes  built-in  carrels  and 
small  cubicles  which  serve  as  spaces  for  individuals  to  study  or  work  by  them¬ 
selves.  If  such  carrels  are  moveable,  they  would  be  classified  as  Equipment. 


Libraries  and  Other  Information  Resource  Centers:  Places  in  which  printed 
and  other  forms  of  mediated  information  is  stored  and  arranged  for  use.  Included 
in  this  element  are  learning  resource  centers,  dial-in  TV  and  other  information 
retrieval  systems,  etc. 
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2.5  Individual  Learning  Spaces 


Description 


«L 

b. 


c. 

d. 


2.6  Libraries  and  Other  Information  Resource  Centers 

a. 

b. 
c 

d. 

e. 


2.7  Other  Facilities 


SOFTWARE 


This  category  of  the  computer-based  training  system  includes  systems 
programming  support,  general  applications  programs,  diagnostic  and  checkout 
software,  utility  programs,  etc. 


EXAMPLE 

3.  SOFTWARE 

3.1  Systems  Programs 
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Systems  Programs:  The  underlying  software  necessary  for  the  control  of  a 
computer  system,  including  operating  systems,  monitors,  executives,  peripheral 
device  interfaces,  etc. 


General  Applications  Programs:  Those  computer  programs  which  are  dependent 
on  data  bases  for  their  use  but  which  can  be  run  on  various  machines  and  have 
usefulness  independent  of  a  specific  course  of  instruction  (e.g.,  SPSS,  BioMed 
Statistical  Packages,  Compilers,  Assemblers  and  Interpreters  such  as  FORTRAN, 
BASIC,  COURSEWRITER,  or  certain  interactive  CAI  functions). 


Diagnostic/Test  Programs:  Special  purpose  programs  used  primarily  to  test 
the  operation  of  computer  system  hardware  components.  Examples  include 
diagnostics  to  locate  damaged  tape  or  disk  surfaces,  faulty  sections  of  computer 
memory,  or  transmission  errors  on  telecommunication  links.  Diagnostic  programs 
often  operate  independently  of  a  computer's  operating  system  software. 
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3.  SOFTWARE 

3.1  Systems  Programs _ 

Description 


a. 

b. 

c 

d. 

e. 


3.2  General  Applications  Programs 


Utility  Programs:  Generalized  data  and  file  manipulation  software  used  to 
create,  copy,  modify  or  delete  machine-readable  data.  Such  utilities  include 
programs  to  copy  tapes,  print  tape  and  disk  files,  convert  files  from  one  data 
representation  (e.g.,  ASCII)  to  another,  etc. 
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3.4  Utility  Programs 


Personnel 

Personnel 

Personnel 

Description 

Hours 

a.  _  _  $  _  $ 


b. 


c. 


d. 


e. 


(To  Part  D  p.  621 

3.5  Other  Computer  Programs 

_  _  $ _  S 

b.  _  _ 

c.  _ _  __ 

d.  _  _ 

e.  _  _  _ 
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INSTRUCTIONAL  SYSTEM  DEVELOPMENT  (ISO)  ACTIVITIES 


This  category  includes  the  costs  incurred  during  the  process  of  developing  the 
instruction.  The  activities  covered  in  this  category  are  described  in  ISD  Phase  I, 
II,  and  III  (Analyze,  Design,  and  Develop).  However,  for  purposes  of  this  speci¬ 
fication,  this  category  does  not  include  the  costs  of  actually  preparing  the  test  • 
items  and  instructional  materials  (see  Category  5 — Instructional  Methods/Materials). 


EXAMPLE 

4.  INSTRUCTIONAL  SYSTEM  DEVELOPMENT  (ISD)  ACTIVITIES 
4.1  Analysis  (Phase  I) 
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Analyze  (Phase  I):  In  this  Phase  an  inventory  of  job  tasks  is  compiled  and 
tasks  are  selected  for  instruction.  Performance  standards  for  tasks  selected  for 
instruction  are  determined  by  interview  or  observation  at  job  sites  and  verified  by 
subject-matter  experts.  The  analysis  of  existing  course  documentation  is  done  to 
determine  if  all  or  portions  of  the  analysis  phase  and  other  phases  have  already  been 
done  by  someone  else  following  the  ISD  guidelines.  As  a  final  step,  the  list  of 
tasks  selected  for  instruction  is  analyzed  for  the  most  suitable  instructional 
setting  for  each  task.  The  outcomes  of  this  phase  are: 


•  A  list  of  tasks  performed  in  a  particular  job. 

•  A  list  of  tasks  selected  for  training. 

•  A  job  performance  measure  for  each  task  selected  for  instruction. 

•  An  analysis  of  any  existing  instruction  to  determine  if  it  is  usable 
in  whole  or  in  part. 

•  Selection  of  the  instructional  setting  for  each  task  selected. 

Design  (Phase  II):  In  this  Phase,  instruction  is  designed  using  the  job 
analysis  information  from  Phase  I.  The  first  step  is  the  conversion  of  each  task 
selected  for  training  into  a  terminal  learning  objective.  Each  terminal  learning 
objective  is  then  analyzed  to  determine  learning  objectives  and  learning  steps 
necessary  for  mastery  of  the  terminal  learning  objective.  Tests  are  to  be  designed 
to  match  the  learning  objectives.  For  purposes  of  this  spool fi. cation,  test  item 
preparation  will  not  he  costed  in  this  subcatajomj .  '/other,  subcatcyorij  ft .  11  is  to 
be  entered  with  that  information.  A  sample  of  students  is  tested  to  insure  that 
their  entry  behaviors  match  the  level  of  learning  analysis.  Finally,  a  sequence  of 
instruction  is  designed  for  the  learning  objectives.  The  outcomes  of  this  phase 
that  are  to  be  accounted  for  in  this  costing  subcategory  are: 

•  A  learning  objective  and  analysis  of  each  selected  task. 

•  A  test  of  entry  behaviors. 

•  The  sequencing  of  all  dependent  tasks. 
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4.  INSTRUCTIONAL  SYSTEM  DEVELOPMENT  (ISD)  ACTIVITIES 
4.1  Analysis  (Phase  I) 


Description 


a. 


Personnel 

Hours 


Non- 

Personnel 

Personnel 

=  Total 

Costs 

Costs 

Costs 
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d. 


< 

(To  Part  D  p.  62) 

4.2  Design  (Phase  II) 


$  $  $ 


b. 

c. 


d. 

e. 


$ 

(To  Part  D  p.  62) 
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Develop  (Phase  III) :  The  instructional  development  phase  begins  with  the 
classification  of  learning  objectives  by  learning  category  so  as  to  identify 
learning  guidelines  necessary  for  optimum  learning  to  take  place.  Determining 
how  instruction  is  to  be  packaged  and  presented  to  the  student  is  accomplished 
through  a  media  selection  process  which  takes  into  account  such  factors  as  learning 
category  and  guideline,  media  characteristics,  training  setting  criteria,  and 
costs.  Instructional  management  plans  are  developed  to  allocate  and  manage  all 
resources  for  conducting  instruction.  Instructional  materials  are  selected  or 
developed  and  tried  out.  For  purposes  of  this  specification,  instructional 
material  preparation  will  not  he  costed  in  this  subcategory .  Rather  subcategories 
5.1  -  5.10  are  to  be  entered  with  that  information.  When  materials  have  been 
validated  on  the  basis  of  empirical  data  obtained  from  groups  of  typical  students, 
the  course  is  ready  for  implementation.  The  outcomes  of  this  phase  that  are  to 
be  accounted  for  in  this  costing  subcategory  are: 

•  The  classification  of  learning  objectives  by  learning  category  and 
the  identification  of  appropriate  learning  guidelines. 

•  The  media  selection  for  instructional  development  and  the  instruc¬ 
tional  management  plan  for  conducting  the  instruction. 

•  The  analysis  of  any  existing  instruction  that  meets  the 
given  learning  objectives. 

•  Field  testing  of  instructional  materials. 
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Year 


1 


4.3  Development  {Phase  III) 


Otter  iption 
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4.4  Other  Instructional  System  Development  (ISD)  Activities 


INSTRUCTIONAL  METHODS/MATERIALS 


Included  here  are  all  forms  of  instructional  methods,  materials,  and  tests 
in  the  computer-based  training  program.  All  print  and  mediated  instruction  is 
included,  as  well  as  specific  applications  computer  programs  when  appropriate. 
Printed  materials  would  include  such  items  as  training  manuals,  instructor  guides 
printouts,  books,  programmed  texts,  etc.  Other  mediated  forms  of  instruction 
include  film,  audio,  audiovisual,  video  and  computer  displays,  etc.  Indicate 
the  estimated  number  of  course  hours  for  each  element  of  instruction  developed. 


EXAMPLE 

5.  INSTRUCTIONAL  METHODS/MATERIALS 
5.9  Computer  Administered  Instructional  Materials 
5.9.2  Simulation 
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Audio:  In  this  subcategory  are  included  instructional  materials  in  forms 

presented  only  for  listening.  Tape  recordings,  phonograph  records,  radio  broad¬ 
casts,  etc.,  comprise  this  subcategory. 


Audiovisual :  Mediated  instructional  materials  which  are  to  be  seen  and 
heard  in  an  Integrated  fashion.  Included  in  this  subcategory  are  slide/tape 
programs,  sound  filmstrips,  sound  motion  pictures,  videotape  and  other  television 
programs,  etc. 

Film/Text/Visual :  Mediated  instructional  materials  on  film  designed  to  be 
viewed  only.  Included  in  this  subcategory  are  slides,  overhead  transparencies, 
film  strips,  silent  motion  pictures,  etc.,  in  which  text  as  well  as  pictures  are 
presented . 

Lecture/Demonstration :  A  portion  of  instruction  in  which  facts  or  concepts 
to  be  learned  are  presented  by  the  instructor,  or  a  skill  to  be  mastered  may  be 
introduced.  Active  student  participation  may  be  elicited  at  critical  points  with 
(1)  questions  or  (2)  by  directing  attention  to  certain  features  demonstrated. 


(To  Part  D  p.  62) 


5.3  Film  Text/Visual 
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(To  Part  D  P  62) 


Group  Discussion/Seminar:  A  small  number  of  students  engaged  in  interactive 
exchange  of  information  about  a  given  topic.  The  instructor  may  or  may  not 
participate. 


Performance/Practice:  Exercises  for  the  student  to  rehearse  a  skill  (1)  with 
feedback  (guided  practice),  or  (2)  without  feedback  (unguided  practice)  on  the 
student's  performance  of  the  skill. 


Tutoring  (peer  or  other):  A  form  of  instruction  involving  one-to-one 
dialogue  between  instructional  agent  (teacher,  fellow  student  or  other)  and 
student . 


t 


Printed  Text/Visual:  Instructional  materials  that  are  printed  on  paper. 
Included  in  this  subcategory  are  books,  training  manuals,  programmed  texts, 
printouts,  forms,  as  well  as  printed  photographs,  etc. 
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5.5  Group  Discussion/Seminar 


5.7  Tutoring  (peer  or  other) 

a. 


Computer-Administered  Instructional  Materials,  instructional  materials  that 
are  prepared  £or  presentation  on  computer  output  devices  such  as  cathode  ray 
tubes,  plasma  panels,  teletypes,  etc.  A  variety  of  instructional  strategies  can 
be  applied  using  the  computer  as  described  below. 

Drill  and  Practice:  Following  presentation  of  a  concept  and  a  test  of 
knowledge  comprehension,  the  student  interacts  with  a  series  of  example  instances. 
The  purpose  of  this  activity  is  to  strengthen  performance  skills  in  the  designated 
concept.  Computer-based  drill  and  practice  permits  repeated  presentations  of  a 
wide  range  of  randomly  generated  or  specially  programmed  instances.  Acceleration 
or  remediation  of  instruction  can  be  made  contingent  on  student  performance. 


Simulation:  An  instructional  situation  which  allows  a  participant  to 
interact  in  a  situation  closely  resembling  an  actual  experience.  Depending  on  the 
sophistication  of  the  computer  system  involved,  the  computer  houses  the  simulation 
rules  and  enables  complex  simulated  decision-making,  monitors  student  progress  and 
maintains  instructional  records  for  subsequent  training  use.  Justification  for 
simulation:  safety  in  performing  dangerous  and  critical  criterial  tasks,  tele¬ 
scoping  real-world  time,  and  permitting  abstraction  of  essential  task  elements 
from  potentially  confusing  total  criterial  environment. 


Games:  A  cooperative  or  competitive  environment  in  which  the  student 
interacts  with  real  or  artificial  participants  to  achieve  specified  goals.  Points 
can  be  offered  for  degree  of  cooperative  or  competitive  achievement.  In  combina¬ 
tion,  simulation  and  games  are  often  used  as  instructional  tools  for  teaching 
problem-solving.  They  are  justified  as  strong  motivating  forces  for  learning. 

In  addition  to  the  automated  functions  ascribed  to  simulation,  the  computer  can 
provide  the  rules  for  real  players  to  interact;  or  it  also  can  be  the  source 
through  artificial  intelligence  of  other  simulated  players. 
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5.9.2  Simulation 


Tutorial:  An  instructional  situation  in  which  the  student  interacts 
with  a  model  of  a  teacher,  the  model  being  resident  in  the  computer.  It  can 
encompass  all  or  some  of  the  instructional  program,  governing  (dialogue) 
sequence  of  materials,  feedback  rules,  strategies  for  simulation,  games,  drill 
and  practice,  or  other  related  instructional  paradigms.  Depending  upon  computer 
sophistication  and  instructional  needs,  automation  of  instructional  prescrip¬ 
tion,  resource  allocation,  and  record-keeping  can  be  partial  or  total. 


Problem-Solving :  One  of  the  most  common  instructional  forms  consists 
of  quantitative  problems  generated  by  stored  algorithms.  The  student  interacts 
by  inputting  step-by-step  process  solutions  at  a  terminal  device  or  the  student 
solves  the  problem  on  paper  and  inputs  his  answer. 


Inquiry:  The  student  forms  questions  which  he  addresses  to  the 
computer  system.  He  may  use  natural  language  or  some  easy-to-learn  subset  of 
the  language.  The  system  processes  the  questions  usually  using  key  words  and 
searching  stored  algorithms  to  provide  an  answer.  This  activity  is  often 
called  information  retrieval. 


42 


(To  p.  45) 


5.9.5  Problem-Solving 

a 

b. 

c. 

dL 

a 


5.9.6  Inquiry 


Specific  Applications  Programs:  Computer  programs  written  to  satisfy  a 
specific  instructional  logic  requirement  in  a  given  course.  These  programs  are 
both  data  and  course  dependent  (such  as  unique  algorithms  written  for  Physics  I 
problems  for  a  particular  instructor).  They  are  usually  written  in  a  CAI  language 
although  they  could  also  be  written  in  a  machine  assembly  language. 


Total  Computer-Administered  Instructional  Materials  Costs:  Enter  total 
costs  obtained  in  subcategories  5.9.1  thru  5.9.8  in  the  spaces  provided.  Sum 
these  costs  to  arrive  at  a  total  cost  of  5.9  subcategory.  Transfer  this  sum 
to  the  appropriate  line  of  the  TCBS  matrix  (Parc  D) . 
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Tests:  In  the  broadest  sense,  tests  are  materials  presented  to  targeted 
individuals  or  groups  in  order  to  measure  their  status  with  respect  to  a  specified 
domain  of  behavior.  The  tests  may  be  direct ,  the  behavior  sampled  is  from  the 
domain,  e.g.,  the  student  must  actually  change  a  tire.  On  the  other  hand,  the 
tests  may  be  indirect,  e.g.,  the  student  describes  by  paper  and  pencil  how  he 
would  change  a  tire.  In  instruction,  these  materials  sample  student  performance 
during  or  at  the  end  of  a  learning  sequence.  They  may  be  used  solely  to  diagnose 
current  level  of  achievement  or  they  may  be  used  in  addition  to  prescribe 
sequences  of  instruction.  If  used  strictly  for  diagnosis,  the  test  items  are 
usually  presented  without  feedback  of  correct  answers. 

Paper  and  Pencil:  These  are  verbal  materials.  Generally  these 
materials  form  an  indirect  measure  of  the  student's  level  of  knowledge  or  compre¬ 
hension.  The  student  writes  answers  to  an  item  by  choosing  from  among  alternatives, 
constructs  a  short  answer  to  complete  a  sentence,  or  writes  an  essay.  The 
inference  is  made  that  achievement  on  these  items  provides  a  valid  and  reliable 
index  for  predicting  achievement  in  the  criterion  environment  (e.g.,  job 
proficiency) . 


Performance  Tests:  Materials  are  those  with  which  the  student  has  to 
do  something,  i.e.,  non-verbal.  They  are  generally  of  two  types:  (1)  intermediate 
or  indirect — in  which  the  student  manipulates  symbols,  patterns,  or  physical 
items  said  to  be  correlated  with  some  desired  performance  not  readily  measured 
in  a  testing  environment;  or  (2)  job-sample  (direct) — in  which  the  student  is 
given  portions  of  job-tasks  to  perform.  These  tasks  are  representative,  critical 
to  job-performance,  or  both. 

I 

I 


5.10  Othar  Instructional  Methods/Materials 


Ducription 


a. 

b. 

c. 

d. 
«. 


5.11  Tests 

5.11.1  Paper  &  Pencil 

a.  _ 

b.  _ 

c.  _ 

d.  _ 

«. 


5.11.2  Performance  Tern 


Computer-Supported  Testing:  If  a  test  is  administered  at  a  display 
device  connected  to  a  computer  (a  terminal),  the  test  is  said  to  be  administered 
on-line.  If  it  is  administered  manually  by  paper  and  pencil  or  by  performing 
tasks  on  devices  or  equipment  not  connected  to  the  computer,  the  test  is  said  to 
be  administered  off-line.  The  support  roles  of  the  computer  can  include  any  or 
all  of  the  following:  (a)  to  store  and  retrieve  banks  of  test  items,  allowing 
efficient  generation  of  numerous  alternate  test  forms;  (b)  to  score  and  print 
out  results  of  the  test  administrations;  (c)  to  maintain  testing  records  for 
purposes  of  test  validation  and/or  student  diagnosis  and  prescription;  and  (d)  to 
administer  tests  on-line. 


Total  Test  Development  Costs:  Enter  total  costs  obtained  in  subcategories 
5.11.1  thru  5.11.4  in  the  spaces  provided.  Sum  these  costs  to  arrive  at  a  total 
cost  of  5.11  subcategory.  Transfer  this  sum  to  the  appropriate  line  of  the 
TCBS  matrix  (Part  D) . 
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V 


5.11.3  Computer -Supported  Testing 


Description 


H 

b. 

t 

d. 

<L 


5.11.4  Other  Tests 

e. 


b. 

c. 

<L 

e. 


Total  Test  Development  Costs  (5.11) 

Paper  &  Pencil  (5.11.1) 

Performance  Tests  (5.11.2) 

Computer  Supported  Testing  (5.11.3) 

Other  Tests  (5.1 1.4) 

Total  5.11 
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SYSTEM  MANAGEMENT/TEST 


This  category  includes  costs  of  the  technical  and  business  management  effort 
expended  in  the  process  of  developing  an  integrated  and  tested  computer-based 
system  prototype.  It  includes  systems  integration/engineering,  program 
management,  and  operational  test  component  costs. 


EXAMPLE 

6.  SYSTEM  MANAGEMENT/TEST 
6.2  Propam  Development 


System  Inteqration/Engineering:  This  subcategory  includes  the  cost  of  the 
effort  expended  in  the  process  of  producing  an  integrated  system.  This  element 
includes  the  cost  of  preparation,  revision,  reproduction  of  drawings,  specifica¬ 
tions,  parts  lists,  test  plans,  testing  procedures,  draft  manuals  and  other 
documentation  which  is  produced  in  support  of  program/project  management  and 
engineering  functions.  It  includes  the  cost  of  gathering,  storing,  reproducing, 
and  disseminating  technical  and  managerial  data,  and  the  cost  of  preparation, 
updating,  and  reproduction  of  publications  such  as  technical  orders,  handbooks, 
and  field  manuals. 

Program  Management:  This  subcategory  contains  costs  for  planning,  directing, 
and  controlling  the  production  of  a  system  and  assuring  that  planning  is  accom¬ 
plished  by  organizations  responsible  for  the  complementary  functions  of  logistics 
and  maintenance  support,  personnel  training,  operational  testing,  activation,  or 
deployment  of  a  system. 

Operational  Test:  This  subcategory  contains  costs  of  system-related  operational 
test  activities  including  cost  of  detailed  planning,  conduct,  support,  data 
reduction,  and  reports  from  such  testing,  as  well  as  hardware  items  which  are 
consumed  in  the  conduct  and  support  of  the  operational  test  program. 
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6.  SYSTEM  MANAGEMENT/TEST 
6.1  System  Integration/Engineering 


19 


$ _ 

_  _  _  „  (To  Pan  D  p  63) 

6.2  Program  Management 


6.4  Other  Direct  Management  Costs 


(To  Pat  t  D  p.  63) 


a. 


$ 


$ 


$ 


OTHER  DIRECT  COSTS 


This  category  contains  elements  which  are  not  covered  in  other  categories. 
Costs  related  to  travel,  supplies,  consultants,  contracts  and  subcontracts, 
etc.,  not  identified  otherwise  are  represented  here. 


employed  in  the  development  of  the  computer-based  training  system. 


Travel :  Included  in  this  subcategory  are  all  transportation  costs,  per 

diem  costs,  etc.,  required  for  system  development. 


Consul tantS :  This  subcategory  includes  costs  associated  with  unique  personal 
services  required  in  the  development  of  the  computer-based  ^raining  system, 
arranged  for  by  the  prime  contractor  or  Government  or  other  sponsor. 
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7.  OTHER  DIRECT  COSTS 


7.1  Supplies 


19 


Description 

No.  Unit  _  Total 

Units  Cost  Cost 

a. 

$  s 

b. 

c. 

d. 

e. 

$ 

(To  Part  O  p.  631 


7.2  Travel 


Description 


Cost 


$ 


7.3  Consultants 


S _ 

(To  Par t  Dp  63) 


Year 


GFE 

(/) 


$ 


Contracts/Subcontracts :  This  subcategory  includes  materials  and  services  to 
be  supplied  by  private  f inns/institutions/organizations  to  assist  in  the  develop¬ 
ment  of  the  computer-based  training  system.  These  are  arranged  or  contracted  for 
by  the  prime  system  contractor,  or  by  the  Government  or  other  sponsor. 


Part  D 

TRAINING  COST  BREAKDOWN  STRUCTURE  (TCBS)  MATRIX 


-meE&IMG  P66B  NO?  FUMED 
BANK 
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Part  D 


TRAINING  COST  BREAKDOWN  STRUCTURE  (TCBS)  MATRIX 


In  this  part  of  the  specification,  a  matrix  is  provided  for  entering 
total  yearly  costs  of  developing  the  computer-based  training  system.  For 
purposes  of  obtaining  total  Development  costs,  the  element  costs  shown  in 
Part  C  are  to  be  summed  for  each  subcategory  and  entered  into  the 
appropriate  row  of  the  TCBS  matrix.  All  costs  should  be  presented  in  the 
year  expended.  As  noted  in  Part  B,  General  Costing  Assumptions  and 
Definitions,  all  costs  should  be  expressed  in  constant  dollars  (using  the 
present  year  as  the  base  year).  Total  costs  for  each  category  are  to  be 
calculated  and  entered  for  each  year.  These  total  category  costs  are  then 
to  be  summed  to  arrive  at  total  Development  costs  for  each  year  covered 
in  this  Phase  of  the  system's  life  cycle. 


-WtCCEBING  PAGE  NOT  FILMED 
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TRAINING  COST  BREAKDOWN  STRUCTURE  (TCBS)  MATRIX 
DEVELOPMENT  PHASE 


1.  Equipment 

1.1  Computer!*) 

1.2  Terminal  (s) 

1.3  Auxiliary  AV  Devices 

1.4  Auxiliary  Memory 

1.5  Local  Interfaces 

1.6  Telephone  Lines 

1.7  Special  Lines 

1.8  Satellites 

1.9  Receivers 

1.10  Power  Generating  Equipment 

1.11  Carrels 

1.12  Other  Equipment 
TOTAL  EQUIPMENT  COSTS 


YEAR 


19  19  19  19  19 


$  $  $  $ _  $ 


2.  Facilities 

2.1  Classrooms  $ _  $ _  $ _  $ _  $ 

2.2  Laboratories  _  _  _  _  _ 

2.3  Large  Group  Inst.  Spaces  _  _  _  _ 

2.4  Offices  _  _  _  _  _ 

2.5  Individual  Learning  Spaces  _  _  _  _  _ 

2.6  Libraries  8i  Other  Info.  Resource  Ctrs.  _  _  _  _ _  _ 

2.7  Other  Facilities  _  _  _  _  _ 

TOTAL  FACILITIES  COSTS  $  $  $  S _ $ 
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3.  Software 

3.1  Systems  Programs  £ 

3.2  General  Applications  Proyams  _ 

3.3  Oiagnostic/Test  Programs  _ 

3.4  Utility  Programs 

3.5  Other  Computer  Programs  _ 

TOTAL  SOFTWARE  COSTS  $ 

4.  Instructional  System  Development 

4.1  Analyze  (Phase  I) 

4.2  Design  (Phase  II) 

4.3  Development  (Phase  III) 

4.4  Other  Instructional  System 
Development  (ISO)  Activities 

TOTAL  INSTRUCTIONAL  SYSTEM 
DEVELOPMENT  COSTS 

5.  Instructional  Methods/Materials 

5.1  Audio  $ 

5.2  Audio/Visual 

5.3  Film  Text/Visual 

5.4  Lecture/Demonstration 

5.5  Group  Discussion/Seminar 

5.6  Performance/Practice 

5.7  Tutoring  (peer  or  other) 

5.8  Printed  Text/Visual 

5.9  CAI  Materials 

5.10  Other  Instructional  Methods/Materials 

5.11  Tests 

TOTAL  INSTRUCTIONAL  METHOOS/ 
MATERIALS  COSTS  $ 


6.  System  Management/Test 

6.1  System  IntegratioryEngineering 

6.2  Program  Management 

6.3  Operational  Test 

6.4  Other  Direct  Management  Costs 

TOTAL  SYSTEM  MANAGEMENT/ 
TEST  COSTS 

7.  Other  Direct  Costs 

(Not  Included  Above) 

7.1  Supplies 

7.2  Travel 

7.3  Consultants 

7.4  Contracts/Subcontracts 
7.6  Other  Direct  Costs 
TOTAL  OTHER  DIRECT  COSTS 


Total  Development  Costs 


Part  E 

TRAINING  EFFECTIVENESS  ASSUMPTIONS  AND  DEFINITIONS 
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Part  E 


TRAINING  EFFECTIVENESS  ASSUMPTIONS  AND  DEFINITIONS 


In  this  section  of  the  specification,  guidance,  definitions,  and 
procedures  are  provided  for  the  measurement  of  training  system  effective¬ 
ness.  A  general  discussion  of  evaluation  and  effectiveness  is  followed 
by  definitions  of  terms  used  in  this  specification.  Effectiveness 
measures  are  described  and  procedures  are  recommended  for  acquiring  this 
information. 

EVALUATION  OF  TRAINING  EFFECTIVENESS 

The  purpose  of  evaluating  training  is  two-fold:  (1)  to  continuously 
improve  the  program  being  evaluated;  (2)  to  decide  if  a  given  training 
program  is  to  be  continued  or  adopted.  Data  needed  for  both  purposes 
should  be  related  to  the  two  sets  of  training  goals  which  exist  for  a 
given  course:  end-of-course  objectives,  and  job  requirements  for  which 
the  student  is  being  trained.  Training  effectiveness  information  should 
be  collected  to  determine  the  degree  to  which  these  training  goals  are 
achieved. 

The  first  type  of  information  should  describe  the  ability  of  a 
course  graduate  to  perform  acceptably  those  tasks  which  the  instructional 
program  claims  to  teach.  In  most  instances,  these  training  effectiveness 
assessments  can  be  made  at  the  school  or  training  sites.  The  second 
type  of  training  effectiveness  data  deals  with  the  discrepancies  between 
graduate  performance  and  job  requirements.  This  information  describes 
whether  or  not  the  instructional  program  teaches  appropriate  subjects  or 
tasks,  and  whether  or  not  the  student  can  transfer  these  capabilities  to 
the  field. 

As  there  are  no  unequivocal  results  from  delayed  types  of  evaluation, 
the  training  effectiveness  of  a  course  can  be  assessed  most  meaningfully 
by  obtaining  the  required  measuresments  (e.g. ,  student  achievement) 
iinnediatly  following  course  completion.  These  measurements  reflect  the 
validity  or  relevancy  of  the  training  only  to  the  extent  that  they  are 
related  to  job  requirements.  If  training  effectiveness  measurements  can¬ 
not  be  taken  immediately  following  course  completion,  then  they  should  be 
obtained  as  soon  as  possible  from  the  job  or  operational  setting. 

•  Evaluation.  Evaluation  has  as  its  major  characteristic  the  deter¬ 
mination  of  value,  worth,  or  merit.  It  has  a  more  general  meaning  than 
effectiveness.  While  training  effectiveness  data  can  be  used  as  the  basis 
for  evaluative  decisions,  all  data  gathered  for  evaluation  purposes  are 
not  necessarily  related  to  effectiveness. 
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•  Formative  Evaluation.  When  evaluation  procedures  are  used  for 
course  improvement  they  are  categorized  as  "formative."  Therefore, 
formative  evaluation  is  that  process  which  validates  instruction  during 
ongoing  initial  program  development.  This  formative  process  may  occur 
during  the  Development  Phase  and/or  the  Operation  and  Maintenance  Phase 
of  the  computer-based  training  system  life  cycle.  Its  purpose  is  to 
diagnose  and  correct  the  weaknesses  of  a  program.  The  results  of  this 
evaluation  are  acted  upon  immediately  in  program  modification.  One  may 
be  concerned  with  the  differential  contribution  of  each  of  the  instruc¬ 
tional  components  to  the  total  effectiveness  of  the  course.  The  training 
effectiveness  of  individual  course  sections  can  be  examined  by  the  use 

of  measures  and  techniques  designed  specifically  for  this  purpose.  (For 
example,  achievement  or  time  meast  res  for  intermediate,  within-course 
criteria.)  These  measures  may  not  be,  and  in  many  cases  should  not  be, 
the  same  as  those  needed  for  judging  the  effectiveness  of  the  total 
course. 

•  Summative  Evaluation.  This  type  of  evaluation  is  performed  for 
the  purpose  of  assessing  a  fully  implemented  training  program  with  respect 
to  its  ability  to  produce  graduates  who  can  perform  to  minimum  standards 
of  performance.  In  this  evaluation,  the  criteria  of  effective  training 
may  be  external  to  the  system.  Only  after  the  course  has  been  established 
as  producing  successful  graduates  can  the  criteria  of  reenlistment,  or 
transfer  to  a  job  or  to  another  course,  be  applied.  In  addition,  a 
summative  evaluation  can  determine  whether  or  not  efficient  and  effective 
use  was  made  of  educational  resources.  Summative  evaluation  should  occur 
after  instructional  development,  improvement,  and  stabilization  of  oper¬ 
ational  and  administrative  activities.  The  results  of  such  an  evaluation 
are  of  primary  concern  to  those  who  will  decide  whether  or  not  a  program 
is  to  be  continued  or  adopted.  It,  therefore,  provides  the  basis  for 
policy  decisions  that  do  not  necessarily  concern  revision  of  a  program  or 
a  product.  Once  the  success  of  the  computer-based  curriculum  is  estab¬ 
lished,  comparisons  can  then  legitimately  be  made  between  it  and  other 
means  of  producing  successful  graduates  (i.e. ,  conventional  lock-step 
instruction  or  other  means  of  self-paced  instruction).  This  type  of  eval¬ 
uation  generally  will  occur  during  the  Operation  and  Maintenance  Phase. 

9  Training  Effectiveness.  This  term  refers  to  the  specific  effects 
that  training  has  on  the  students  who  receive  instruction.  These  effects 
are  usually  measured  in  terms  of  the  time  it  took  students  to  reach  given 
training  goals/objectives,  and/or  in  the  levels  of  achievement  reached. 
Training  effectiveness  data  should  be  collected  to  determine  the  degree  to 
which  a  training  course  or  system  achieves  its  established  goals.  This 
can  only  be  done  by  evaluating  the  graduates  of  the  training  program. 
Effectiveness  relates  to  the  broader  concept  of  evaluation  in  that  the 
effects  of  training  are  judged  as  desirable  or  undesirable  by  reference 
to  the  goals  and  objectives  of  the  training  program. 

•  Training  Efficiency.  This  term  refers  to  the  amount  of  knowledge 
or  skill  gained  in  relation  to  the  amount  of  time  spent  in  achieving  that 
gain.  Thus,  for  example,  a  student  who  attains  mastery  within  one  hour 
is  twice  as  efficient  as  the  student  who  reaches  mastery  in  two  hours. 
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Course  efficiency  refers  to  the  number  of  students  who  successfully 
achieve  training  goals  per  unit  time  or  within  given  training  resources. 

TRAINING  EFFECTIVENESS  MEASURES 

Several  types  of  measurements  for  assessing  training  effectiveness 
are  possible.  Foremost  among  these  are  direct  measures  of  student 
achievement  which  reflect  attainment  of  course  goals  and  learning 
objectives.  These  measurements  can  be  in  terms  of  test  scores,  gain 
scores,  time  to  achieve  mastery,  number  of  objectives  mastered,  etc. 

The  specific  measurements  selected  should  correspond  to  the  requirements 
as  stated  in  each  course's  training  objectives. 

Effectiveness  measures  may  be  either  short  term  or  long  term.  Short¬ 
term  measures  can  reflect  the  attainment  of  intermediate  goals  within  a 
course,  as  well  as  final  goals,  such  as  end-of-course  achievement.  Long¬ 
term  measures  are  designed  to  assess  the  transfer  of  the  effects  of 
training  to  other  environments. 

In  the  ideal  case,  the  effectiveness  of  a  training  program  should  be 
evaluated  from  the  viewpoint  of  long-term  as  well  as  short-term  criteria 
measures.  However,  many  intervening  factors  and  events,  as  well  as 
attitude  changes,  can  take  place  and  effect  the  long-term  measures, 
totally  unrelated  to  the  course  of  instruction. 

As  it  is  usually  impractical  to  account  for  these  intervening  factors, 
the  validity  of  long-term  measures  decreases  the  longer  these  measures 
are  separated  in  time  from  the  training  that  is  being  evaluated. 

Therefore,  for  most  uses  of  this  specification,  the  training  effec¬ 
tiveness  of  a  course  should  be  assessed  immediately  upon  completion  of 
that  course.  Tests  administered  at  that  time  are  the  most  direct  and 
relatively  unconfounded  measures  of  training  effectiveness. 

Other  measurements  that  might  be  taken  which  could  reflect  attain¬ 
ment  of  training  program  goals  can  be  categorized  as:  attitude  and  other 
affective  measures;  ratings  of  achievement  (student  and/or  instructor); 
and  unanticipated  secondary  effects.  It  is  important  to  look  at  as  many 
meaningful  outcomes  of  a  program  as  possible  even  if  they  were  not  antici¬ 
pated  in  the  program's  stated  goals  and  objectives.  It  is  highly  probable 
that  if  the  course  goals  and  objectives  were  stated  completely ,  an 
absence  of  negative  side  effects  would  be  desired  and,  in  many  cases, 
insisted  upon  by  the  training  developers. 

To  describe  the  overall  training  effectiveness  of  a  course  or 
training  system,  the  individual  student  measurements  need  to  be  accumu¬ 
lated  and  summarized.  Measures  of  central  tendency  (e.g. ,  means, 
medians,  modes),  ranges,  frequencies  and  other  descriptive  statistical 
indices  can  be  used  for  this  purpose. 

Once  measurements  have  been  made  and  summarized,  the  resulting  data 
must  be  compared  with  standards  or  criteria  which  reflect  the  training 
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goals.  Thus,  the  units  of  measurement  must  correspond  to  the  units  of 
the  criterion  for  each  training  effectiveness  dimension. 

•  Absentee  Rates.  This  measure  refers  to  the  average  percentage 
of  students  who  are  not  present  for  training  on  any  given  day  for  each 
course. 

•  Accuracy  Scores.  Accuracy  scores  indicate  the  level  of  achieve¬ 
ment  reached  within  certain  tolerances  on  specific  scales.  These  scores 
reflect  the  quality  or  precision  of  performance.  Such  scores  can  be 
described  in  terms  of  the  number  of  positive  instances  (correct  items)  or 
the  number  of  errors  made  on  a  given  scale. 

•  Achievement  Measures.  These  measures  either  reflect  student 
achievement  or  within-course  enabling  objectives  or  assess  end-of-course 
proficiency.  In  order  for  this  kind  of  assessment  to  be  possible,  the 
objectives  must  be  explicitly  stated  and  translated  into  performance  or 
behavioral  terms.  Examples  of  such  achievement  measures  would  be  scores 
on  criterion-referenced  tests  related  to  objectives,  or  gain  scores 
taking  into  account  the  initial  proficiency  level  of  each  student.  Addi¬ 
tional  scores  that  can  be  used  to  describe  a  student's  level  of  achieve¬ 
ment  are:  nunber  of  objectives  mastered,  skill  level  attained,  etc.  All 
of  these  would  be  objective-based  achievement  scores;  Normative  (norm- 
referenced)  test  scores,  or  a  student’s  class  standing  are  achievement 
scores  that  are  used  when  comparing  each  student's  performance  with  that 
of  the  class  as  a  whole.  If  achievement  scores  are  used  to  compare  the 
effectiveness  of  different  training  programs,  a  considerable  effort 
should  be  undertaken  to  identify  the  differences  that  may  exist  in 
training  objectives  and/or  student  population  characteristics. 

Objective  achievement  or  performance  tests  are  the  best  means  of 
measuring  training  effectiveness.  These  tests  contain  items  or  tasks 
which  cannot  be  answered  or  performed  correctly  without  the  benefit  of 
training.  Thus,  they  reflect  directly  on  training  effectiveness. 

Paper- and- pencil  taste  can  be  used  to  acquire  data  on  achieved 
level  of  job  knowledge.  Such  tests  are  used  for  obtaining  data  on  student 
proficiency  at  recall  or  recognition  of  facts,  principles,  or  procedures, 
and  in  the  application  of  such  knowledges. 

Performance  tests  consist  of  systematic  observation  and  scoring  of 
the  performance  of  students  in  test  situations  which  sample  actual  job 
operations  and  requirements.  This  can  occur  in  either  the  actual  or 
simulated  job  environment.  Performance  tests  are  most  appropriate  for 
evaluating  the  effectiveness  of  technical  skill  training. 

Computer-supported  tests  consist  of  those  tests  that  are  administered 
by  the  computer  and/or  require  the  student  to  input  responses  on-line. 

Subj ective  measures  yield  data  based  on  options  about  the  proficiency 
of  the  student /graduate,  rather  than  direct  measurements  of  skill  or 
knowledge.  Though  not  as  informative  and  useful  as  objective  measures. 
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subjective  measures  and  instruments  often  provide  the  only  practical  means 
for  assessing  training  effectiveness.  Questionnaires,  cheakliats,  rating 
scales,  and  interviews  are  often  used  when  testing  is  inappropriate  or 
not  feasible. 

t  Attitude  Scales.  These  are  subjective,  indirect  measures  of 
training  effectiveness.  They  are  designed  to  elicit  the  opinions  of 
students  regarding  their  positive  or  negative  feelings  about  the  course, 
method  of  instruction,  or  other  related  matters.  They  can  be  used  as 
indicators  of  the  emotional  states  of  students,  instructors,  or  others. 
These  measures  can  point  to  possible  training  problems  and  provide  a  basis 
for  interpreting  the  proficiency  scores  attained  in  the  course. 

•  Attrition  Rates.  This  measure  refers  to  the  percentage  of  students 
who  fail  to  meet  the  within-course  or  end-of-course  criteria  and  are 
dropped  from  the  training  program.  It  also  includes  those  students  who 
voluntarily  leave  the  training  program  for  reasons  other  than  academic 
failure. 

•  Cri teria.  Prior  to  making  any  cost-effectiveness  decision  (i.e., 
prior  to  a  decision  that  the  effectiveness  attained  in  a  given  course  of 
instruction  was  worth  the  investment  in  dollars),  the  decision  maker 
must  set  effectiveness  criteria.  Priorities  should  be  established  among 
the  various  measures  as  to  their  importance  in  assessing  training  effec¬ 
tiveness.  Whether  the  measures  are  objective-related  or  norm-referenced, 
the  achievement  scores  of  individual  students  or  graduates  must  be  appro¬ 
priately  summarized  to  describe  overall  course  effectiveness.  If  means 
or  other  measures  of  central  tendency  are  used  to  reflect  training 
effectiveness,  then  similar  criterion  values  need  to  be  established. 

•  Criterion- Referenced  Tests.  This  type  of  effectiveness  measure¬ 
ment  is  used  when  performance  by  a  student  is  to  be  evaluated  against 
empirically  or  rationally  established  standards.  The  performance  standard 
is  usually  derived  from  an  analysis  of  the  job  skills  for  which  training 
is  designed.  There  can  be  paper-and-pencil ,  as  well  as  performance- 
based  criterion-referenced  tests.  Criterion-referenced  tests  are  usually 
contrasted  with  norm-referenced  measures  in  which  one  is  interested  in 
how  individuals  compare  with  each  other.  Criterion-referenced  tests  can 
indicate  the  degree  of  achievement  of  a  student  by  specifying  the  number 
or  percentage  of  objectives  attained. 

•  Gain  Scores.  These  scores  refer  to  the  difference  between 
scores  on  a  pre-test  taken  prior  to  a  course  of  instruction  and  scores  on 
an  equivalent  post-test  following  a  given  course  of  instruction.  The  gain 
(difference  between  pre-  and  post-tests)  is  said  to  be  the  measure  of 
effectiveness.  One  weakness  in  this  type  of  measure  stems  from  the  fact 
that  the  absolute  amount  that  can  be  gained  by  a  student  is  a  function 

of  his  initial  level  in  the  skill  being  measured.  Therefore,  it  is  not 
a  sufficient  measure  unless  students  enter  the  program  with  relatively 
low  levels  of  the  skill  for  which  they  are  being  trained. 
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•  Mastery.  Generally,  this  term  means  achievement  of  all  require¬ 
ments  to  exhibit  expertise  in  a  given  course  of  instruction.  In  practical 
terms,  it  is  often  used  to  mean  achievement  of  the  required  number  of 
objectives  at  a  minimally  acceptable  level  of  performance.  If  instruc¬ 
tional  decisions  are  made  on  a  "go/no-go"  basis  (achievement  or  not)  per 
course  objective,  then  mastery  is  defined  as  meeting  the  required  number 
of  "go"  decisions  in  order  to  complete  the  course. 

•  Norm- Referenced  (Normative)  Tests.  These  tests  are  generally  used 
to  indicate  relative  standing  among  members  of  a  given  student  population. 
They  are  often  set  in  opposition  to  criterion- referenced  tests  which  are 
used  to  measure  individual  student  achievement  against  established 
standards. 

•  Rati nqs .  Ratings  are  indirect,  subjective  measures  of  effective¬ 
ness.  While  ratings  have  the  advantage  of  being  easily  obtained,  they 
often  have  low  reliability  and  validity.  As  ratings  are  indirect 
measures  of  performance,  they  are  less  valid  than  direct  measures.  It  is 
often  difficult  to  obtain  consistent  ratings  over  time  from  the  same 
individual,  and  from  several  raters  at  the  same  time  (inter-rater 
reliability)  without  training  the  raters. 

•  Recycles  (Washbacks).  This  term  refers  to  the  number  or  percentage 
of  students  who  are  forced  to  go  through  all  or  a  portion  of  a  course 

more  than  once,  based  upon  failure  to  achieve  that  course  section's  required 
objectives  in  a  specified  amount  of  time.  These  students  are  then  recycled 
through  the  instruction  and  given  an  opportunity  to  take  the  criterion 
test(s)  again. 

t  Reenlistment  Rates.  This  long-term  measure  refers  to  the  percent¬ 
age  of  personnel  who  voluntarily  extend  beyond  their  initial  tour  of  duty 
for  another  period  of  military  service.  However,  reenlistment  can 
be  a  function  of  many  variables  extraneous  to  the  training  program. 

•  Rel i abi 1 i ty .  Measures  of  training  effectiveness  should  be  selected 
on  the  basis  of  their  potential  for  obtaining  reliable  and  valid  data. 
Reliability  refers  to  the  consistency,  or  replicability  of  the  data  that 
are  collected.  It  is  not  possible  to  determine  the  validity  of  data  unless 
the  data  are  reliable. 

•  Speed  Scores.  This  term  refers  to  the  rate  of  acceptable  task 
performance.  If  time  or  speed  of  performance  is  the  standard  for  certain 
training  objectives,  then  a  measure  of  training  effectiveness  must  reflect 
this  criterion  dimension.  If  a  task  must  be  performed  in  a  given  time,  a 
possible  effectiveness  measure  to  use  could  be  the  number  of  students  who 
perform  acceptably  in  that  time.  Another  speed  score  could  be  the  number 
of  steps  or  activities  in  a  task  that  are  completed  by  a  certain  percent¬ 
age  of  students  by  a  given  time.  In  either  case,  the  scores  used  to 
measure  training  effectiveness  must  reflect  the  training  objectives  and 
criteria. 
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•  Throughput.  This  measure  refers  to  the  number  of  students 
passing  a  within-course  requirement  per  unit  time,  or  the  nuntier  of 
students  completing  the  course  per  unit  time.  In  this  sense,  it  is  an 
index  of  the  efficiency  of  a  training  program. 

•  Time  Measures 


Course  Time.  This  measure  refers  to  the  total  nuntier  of  hours, 
days,  weeks  or  other  time  units  a  student  spends  in  a  given  course  in 
order  to  reach  mastery  of  the  required  objectives.  It  represents  the  sum 
of  training  time  and  testing  time — see  below. 

Training  Time.  This  measure  represents  the  time  actually  spent  in 
learning  the  subject  matter  and  practicing  the  skills  to  be  acquired.  It 
does  not  contain  time  devoted  to  non-academic  subjects,  in-  and  out- 
processing,  and  other  activities  unrelated  to  the  instruction.  Training 
time  should  include  any  self-study,  remedial  or  recycle  time,  even  if 
these  times  are  not  officially  prescribed. 

Testing  Time.  This  measure  represents  the  time  spent  in  taking 
criterion  tests.  It  includes  all  the  time  spent  in  taking  tests  that  are 
graded — tests  that  determine  if  a  student  has  attained  the  training 
objectives  of  the  course.  Testing  time  does  not  include  time  spent  in 
skill  practice,  practical  exercises,  or  quizzes  that  are  part  of  the 
instruction  or  training  time. 

•  Transfer.  A  transfer  measure  of  effectiveness  is  any  score  (time, 
trials,  errors,  etc.)  obtained  on  tasks  performed  subsequent  to  course 
completion.  The  transfer  measures  are  taken  after  the  student  attains  the 
required  objectives  within  a  course.  These  measures  can  be  relatively 
immediate  and  refer  to  achievement  in  another  course.  Or,  they  can  refer 
to  initial  job  performance  following  the  completion  of  a  course.  One 
possible  index  of  transfer  effects  that  could  be  used  is: 

Percent  Transfer.  The  criterion  value  (time,  trials,  errors,  etc.) 
that  is  reduced  as  a  result  of  relevant  prior  training.  This  is  expressed 
as  a  percentage  of  what  the  criterion  value  would  be  without  such  prior 
training. 


On  a  longer  term  basis,  transfer  can  refer  to  overall  job  effec¬ 
tiveness  in  terms  of  such  performance  measures  as  Army  SQTs,  or  the  Air 
Force's  ATC  Field  Follow-Up  measures.  Other  job  performance  data  can 
include:  supervisor  ratings,  work  productivity  measures,  absenteeism  and 
tardiness  rates,  etc.  Transfer  can  also  refer  to  the  relationship  of 
course  achievement  to  attitudes.  For  example,  it  could  refer  to  effects 
of  course  success  or  failure  on  reenlistment  rates,  career  development 
pattern,  or  general  attitudes  toward  the  Service. 

•  Unanticipated  Secondary  Effects.  Sometimes  used  as  training 
effectiveness  measures  are  serendipitous  effects  that  are  not  directly 
related  to  stated  or  established  training  goals.  A  positive  side  effect, 
for  example,  may  be  increased  independent  study  by  a  graduate  as  a  result 
of  his  positive  experience  within  a  course.  An  example  of  a  negative 
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side  effect  might  be  an  Increase  In  AWOL  or  disciplinary  problems  due  to 
the  lack  of  sufficient  sig>ervision. 

•  Validity.  Training  effectiveness  data  are  valid  if  they  accurately 
reflect  the  degree  of  achievement  of  course  goals.  The  instruments  used 
to  measure  effectiveness  must  be  carefully  selected  in  order  that  validity 
and  reliability  of  the  data  are  maximized.  This  is  particularly  difficult 
with  subjective  measures  such  as  attitude  scales  or  ratings.  Also,  valid 
data  are  obtained  only  if  the  person  being  evaluated  reacts  in  a  normal 
manner  when  tested.  Objective  data  gathering  measures  (e.g.,  performance 
tests)  are  better  at  eliciting  "normal"  and,  therefore,  valid  reactions, 
than  are  subjective  measures  (e.g.,  ratings). 


Part  F 

WITHIN-COURSE  TRAINING  EFFECTIVENESS  DATA 
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INSTRUCTIONS  FOR  ENTERING 
WITHIN-COURSE  TRAINING  EFFECTIVENESS  DATA 


In  this  part  of  the  specification,  a  recording  form  is  provided  for 
you  to  enter  data  on  training  effectiveness  for  each  of  the  major  Sections 
of  the  Course.  A  Section  is  defined  as  the  largest  part  01  a  course  ror 
which  criterion  tests  (based  on  objectives)  or  other  achievement  measures 
have  been  established.  A  Section  may  be  the  equivalent  of  an  Annex,  Unit, 
Lesson,  Task,  etc. ,  depending  upon  the  terminology  used  in  a  given  Service, 
school,  or  curriculum.  MAKE  A  COPY  OF  THE  RECORDING  FORM  ON  PP.  77  AND  79 
FOR  EACH  MAJOR  COURSE  SECTION.  These  data  will  then  be  summarized  on  the 
recording  form  presented  on  p.  83.  The  training  effectiveness  of  the 
Course  as  a  whole  can  be  estimated  from  this  summary,  although  the  end-of- 
course  data  required  in  Part  G  are  preferred  for  this  purpose. 


© 


Print  title  of  Course. 


© 


Print  title  of  Course  Section.  Use  one  recording  form  for  each 
Section.  As  the  numbers  of  Sections  will  vary  with  each  Course,  you 
are  to  enter  the  Section  Numbers  in  sequence. 


©  Enter  the  average  training  time  spent  by  students  in  each  instruc¬ 
tional  method/mode  used  in  this  Course  Section.  This  will  permit  an 
analysis  of  a  Course's  time  scores  with  regard  to  each  mode/medium 
of  instruction.  This  information  should  represent  the  time  actually 
spent  in  learning  the  subject  matter.  It  should  not  contain  course 
time  that  is  unrelated  to  this  function  (e.g. ,  non-academic  subjects, 
in-  and  out-processing,  etc.).  In  the  cases  where  training  times  are 
officially  prescribed,  care  should  be  taken  to  include  additional 
self-study  and  recycle  time,  if  relevant.  Enter  average  training 
times  to  the  nearest  tenth  of  an  hour,  along  with  the  nunfcer  of 
students  (N)  from  whom  the  data  were  obtained.  Then  add  these  times 
together  to  obtain  the  Average  Training  Time  for  this  Section  0  • 


4)  Enter  the  average  time  spent  by  students  taking  the  criterion  tests 

within  this  Course  Section.  Do  not  include  the  time  spent  in  quizzes 
and  practical  exercises  or  skill  practice  that  are  part  of  the  instruc¬ 
tion.  Rather,  that  time  should  be  included  in  training  time  above. 

This  entry  should  include  the  time  spent  in  tests  that  are  graded — 
tests  that  are  designed  to  assess  whether  or  not  students  have  attained 
the  instructional  objectives  of  this  Course  Section.  Enter  the 
average  testing  times  (to  the  nearest  tenth  of  an  hour)  in  the  space 
next  to  the  appropriate  test  form  category,  along  with  the  number  of 
students  (N)  from  whom  the  data  were  obtained.  Then  add  these  times 
together  to  obtain  the  Average  Testing  Time  for  this  Section  ©  • 


© 


Sum  the  Average  Training  and  Testing  Times  ©  and  ©  to  obtain 
the  average  time  spent  In  this  Section  of  the  Course. 
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TRAINING  EFFECTIVENESS  DATA:  WITHIN-COURSE  MEASURES 


(7)  Course _ 

(2)  Section  No. _ 

Time  Measures 

AVERAGE  TRAINING  TIME 

Hours  N 


Audio 

Audiovisual 

Film/Text/Visual 

Lecture/Demonstration 

Group  Discussion/Seminar 

Performance/Practice 

Tutoring 

Printed  Text/Visual 
Other 


Hours  N 


CAI:  Drill  &  Practice  _  _ 

Simulation  _  _ 

Games  _  _ 

Tutorial  _  _ 

Problem-Solving  _  _ 

Inquiry  _  _ 

Specific  Applications  Programs  _  _ 

Other  CAI  Materials  _  _ 

CAI  Materials  (Total)  _ 

®  Average  Training  Time  in  Section  No. _ 

(4)  AVERAGE  TESTING  TIME 

Hours 

Paper  and  Pencil  Tests  _ 

Performance  Tests  _ 

Computer-Supported  Tests  _ 

Other  Tests  _ 

®  Average  Testing  Time  in  Section  No. _ 


®  AVERAGE  SECTION  TIME 

Make  as  many  copies  as  required  for 
the  number  of  Sections  of  the  Course. 


(To  Summary  Form) 
(page  83) 


N 


(To  Summary  Form) 
(page  83) 


(To  Summary  Form) 
(page  83) 
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Enter  the  criterion  test  results  used  to  measure  achievement  of  this 
Section’s  instructional  objectives.  These  data  should  represent  the 
scores  and  results  obtained  from  students'  first  attempts  at  the 
criterion  tests.  (Identify  the  specific  tests  being  reported.) 

(a)  If  accuracy  or  speed  scores  are  used  to  measure 
performance,  enter  the  average  scores  in  the  appro¬ 
priate  space. 

(b)  In  those  Course  Sections  where  the  achievement  measu'e 
is  the  number  of  objectives  attained,  enter  the  average 
number  of  objectives  achieved  in  the  appropriate  space. 

(c)  Enter  the  percentage  of  students  who  pass  this  Section's 
criterion  tests  on  their  first  attempt. 

(d)  If  the  measures  of  achievement  used  in  this  Section  are 
gain  scores  (differences  between  pre-  and  post-tests), 
enter  the  average  gain  score  in  the  appropriate  space. 


Enter  the  average  number  of  attempts  taken  to  pass  the  criterion  tests 
on  this  Course  Section. 


In  this  part  of  the  form,  enter  other  data  deemed  important  for 
describing  the  training  effectiveness  of  this  Section  of  the  Course. 
Affective  data  (e.g. ,  student  attitudes),  indirect  measures  such  as 
instructor  ratings,  as  well  as  other  measures  used,  should  be 
described  and  the  findings  listed. 
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TRAINING  EFFECTIVENESS  DATA:  WITHIN-COURSE  MEASURES  (Continued) 

Achievement  Measures 


SECTION  No. 


© 


-  AVERAGE  CRITERION  TEST  RESULTS  (First  Attempt) 


(a) 

(b) 

(c) 

(d) 

Number 

of 

Tests 

Accuracy /Speed 

No.  Objectives 

Percent 

Gain  Scores 

Attempts 

Scores 

Passed 

Students  Passed 

© 


Paper  and  Pencil:. 


Performance: 


Computer-Supported : 


Other: 


SECTION  No. 


(To  Summary  Form) 
(page  83) 


© 


-  Other  Effectiveness  Measures 


Recycle  Rate. 


Instructor  Ratings 


Student  Attitudes 


Other 


(To  Summary  Form) 
(page  83) 


Make  as  marry  copies  as  required  (or 
the  number  of  Sections  of  the  Course. 
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INSTRUCTIONS  FOR  ENTERING  WITHIN-COURSE 
TRAINING  EFFECTIVENESS  SUMMARY  DATA 


In  this  form  you  are  to  summarize  the  within-course  training 
effectiveness  data  listed  on  the  preceding  forms.  If  only  within- 
course  data  are  available,  these  totals  can  represent  the  overall 
effectiveness  of  this  Course. 


Print  title  of  Course. 

©  Enter  the  Section  Number  in  the  appropriate  space.  If  you  need  more 
than  one  line  to  enter  the  summary  data  for  a  given  Section,  put  that 
nunfcer  in  all  the  applicable  spaces. 


Enter  the  Average  Training  Time  spent  in  each  Section  of  the  Course 
in  the  spaces  provided.  Copy  the  ©  times  from  the  preceding 
forms. 


Enter  the  Average  Testing  Time  spent  in  each  Section  of  the  Course  in 
the  spaces  provided.  Copy  the  0)  times  from  the  preceding  forms. 


Enter  the  Average  Section  Time  in  the  spaces  provided.  Copy  these 
times  0  +  ©  from  the  preceding  forms. 


©  Enter  the  results  from  first  attempts  at  the  criterion  test  in  each 
Section  of  the  Course.  These  data  indicate  the  level  of  performance 
achieved  in  each  Section.  In  some  Sections,  only  one  type  of  achieve¬ 
ment  test  data  may  be  listed.  In  these  cases,  copy  the  data  into  the 
spaces  provided.  If  more  than  one  test  is  used  to  measure  criterion 
performance,  determine  each  student's  grade  in  the  Section  by  weight¬ 
ing  and  combining  the  separate  test  scores.  Enter  this  weighted 
average  score  or  Section  grade  in  the  summary  form.  If  it  is  not 
possible  to  combine  such  scores,  enter  them  on  separate  lines — thereby 
using  more  than  one  line  per  course  Section. 


Enter  the  average  number  of  attempts  to  pass  the  criterion  test(s) 
for  each  Section  of  the  Course. 


Enter  any  other  training  effectiveness  data  obtained  in  each  Course 
Section  in  the  spaces  provided. 
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TRAINING  EFFECTIVENESS  DATA  SUMMARY 
WITHIN-COURSE  MEASURES 


Accuracy  / 
Speed 
Scores 


(Average) 


Achievement 


Number 

Objectives 

Passed 

Percent 

Students 

Passed 

Gain 

Scores 

Number  of 
Attempts 

Recycle 

Rate 

Instructor 

Ratings 

Student 

Attitude 

(Total)  (Average)  (Average) 


Make  as  many  copies  at  required 
for  the  number  of  Courses. 


(Average) 


(Averages) 


1  . 


Part  G 

END-OF-COURSE  TRAINING  EFFECTIVENESS  DATA 
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INSTRUCTIONS  FOR  ENTERING 
END-OF-COURSE  TRAINING  EFFECTIVENESS  DATA 


In  this  part  of  the  specification,  recording  forms  are  provided  for 
you  to  enter  data  on  training  effectiveness  for  the  Course  as  a  whole. 

A  Course  may  be  self-paced  and  achievement  criteria  fixed,  or  Course 
time  may  be  fixed,  and  achievement  allowed  to  vary,  or  there  may  be  a 
combination  of  both  situations.  End-of-course  training  effectiveness 
data  can  be  entered  on  this  form  for  any  of  these  cases.  MAKE  A  COPY  OF 
THE  RECORDING  FORM  ON  PAGES  87,  89,  AND  91  FOR  EACH  COURSE  CONSIDERED. 

If  only  within-course  data  (by  Section)  are  available,  use  the  summarized 
data  in  the  form  at  the  end  of  Part  F  and  go  to  Part  H  for  further  cost- 
effectiveness  analysis. 


© 


Print  title  of  Course. 


©  Enter  the  average  training  time  spent  by  students  in  each  instructional 
method /mode  used  in  this  Course.  These  time  data  should  be  the  sums 
of  the  time  data  listed  in  Part  F  for  each  Section  of  this  Course. 

This  information  should  represent  the  time  actually  spent  in  learning 
the  subject  matter.  It  should  not  contain  Course  time  that  is 
unrelated  to  this  function  (e.g. ,  non-academic  subjects,  in-  and  out- 
processing,  etc.).  In  the  cases  where  these  times  are  officially 
prescribed,  care  should  be  taken  to  include  additional  self-study 
and  recycle  time,  if  relevant.  Enter  average  training  times  to  the 
nearest  tenth  of  an  hour,  along  with  the  number  of  students  (N)  from 
whom  the  data  were  obtained.  Then  add  these  times  together  to  obtain 
the  Average  Training  Time  in  this  Course  ®  • 


© 


Enter  the  average  time  spent  by  students  taking  the  criterion  tests 
within  this  Course.  Do  not  include  the  time  spent  in  quizzes  and 
practical  exercises  that  are  part  of  the  instruction.  Rather,  that 
time  should  be  included  in  above.  This  entry  should  include 

the  time  spent  in  tests  that  are  graded — tests  which  are  designed  to 
assess  whether  or  not  students  have  attained  the  instructional 
objectives  of  this  Course.  These  times  should  include  both  within- 
course  and  end-of-course  tests.  The  time  data  should  be  the  sums  of 
the  time  data  listed  in  Pari  F  for  each  of  the  Sections  in  this 
Course ,  PLUS  the  time  spent  by  the  students  in  the  final  or  end-of- 
course  criterion  tests.  Enter  the  average  testing  times  (to  the 
nearest  tenth  of  an  hour)  in  the  space  beside  the  applicable  test 
form  category,  along  with  the  number  of  students  (N)  from  whom  the 
data  were  obtained.  Then  add  these  times  together  to  obtain  the 
Average  Testing  Time  in  this  Course  ©  • 


© 


Sum  the  Average  Training  and  Testing  Times  ©  and  to  obtain 

the  Average  Course  Time. 


TRAINING  EFFECTIVENESS  DATA:  END-OF-COURSE  MEASURES 


Course. 


Time  Measures 


AVERAGE  TRAINING  TIME 


Audiovisual 

Film/Text/Visual 

Lecture/Demonstration 

Group  Discussion/Seminar 

Performance/Practice 

Tutoring 

Printed  Text/Visual 


GAt:  Drill  &  Practice 


Problem-Solving 


Specific  Applications  Programs 


Other  CAI  Materials 


CAI  Materials  (Total) 


Average  Training  Time  in  Course 


AVERAGE  TESTING  TIME 


Paper  and  Pencil  Tests 
Performance  Tests 
Computer  Supported  Tests 
Other  Tests 

Average  Testing  Time  in  Course 


(4)  AVERAGE  COURSE  TIME 


®*® 


Make  as  many  copies  as  required 
for  the  number  of  Courses, 


■SP 


Enter  the  final  criterion  test  results  used  to  measure  achievement 
of  this  Course's  instructional  objectives.  These  data  should  repre¬ 
sent  the  scores  and  results  obtained  from  students'  first  attempts 
at  the  end-of-course  criterion  tests. 

(a)  If  accuracy  or  speed  scores  are  used  to  measure 
performance,  enter  the  average  scores  in  the  appro¬ 
priate  spaces. 

(b)  In  those  Courses  where  the  achievement  measure  is  the 
number  of  objectives  attained,  enter  the  average 
number  of  objectives  achieved  in  the  appropriate  space. 

(c)  Enter  the  percentage  of  the  students  who  pass  this 
Course's  end-of-course  criterion  tests  on  their  first 
attempt. 

(d)  If  the  measures  of  achievement  used  in  this  Course  are 
gain  scores  (differences  between  pre-  and  post-tests), 
enter  the  average  gain  score  in  the  appropriate  space. 


Enter  the  average  nunfoer  of  attempts  taken  to  pass  the  end-of- 
course  criterion  test(s). 


In  this  part  of  the  form,  enter  other  summarized  data  deemed  impor¬ 
tant  for  describing  the  training  effectiveness  of  this  Course. 
Attrition  and  absentee  rates,  affective  data  (e.g. ,  student  atti¬ 
tudes),  indirect  measures  such  as  instructor  ratings,  as  well  as 
other  measures  used,  should  be  described  and  the  findings  listed. 
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TRAINING  EFFECTIVENESS  DATA:  END-OF-COURSE  MEASURES  ( Continued } 


Achievement  Measures 


© 

© 

|  PINAL  CRITERION  TEST  RESULTS  (First  Attempt) 

Number 

Tests 

(a) 

Accuracy /Speed 
Scores 

(b) 

No.  Objectives 
Passed 

(c) 

Percent 

Students  Passed 

(d) 

Gain  Scores 

of 

Attempts 

Paper  and  Pencil: 


Performance: _ 

Computer-Supported : 

Other: _ _ 


Other  Effectiveness  Measures 

Absentee  Rate _  Recycle  Rate _ 

Attrition  Rate _ 

Instructor  Ratings  Student  Attitudes 


Other 


Transfer  Measures 


Enter  actual  or  estimated  data  that  reflect  the  transfer  effects 
attributed  to  the  training  provided  In  this  Course.  In  all  cases, 
enter  average  values.  This  Information  can  come  from  subsequent 
course  records,  job  performance  tests,  field  feedback  records,  etc. 

Try  to  isolate  the  influence  of  training  in  this  Course  from  other 
intervening  events  (e. g. ,  job  experience,  etc.).  Collect  these  data 
as  soon  after  Course  completion  as  possible.  If  data  are  estimates, 
place  (E)  after  the  entry. 


TRAINING  EFFECTIVENESS  DATA:  END-OF-COURSE  MEASURES  (Continued) 

(?)  Transfer  Measures 

Time  to  Complete  Subsequent  Course _ 

Achievement  Score  in  Subsequent  Course _ 

Job  Performance 

Time  to  be  Job  Qualified _ 

Test  Score(s) _ _ 

Work  Productivity  Measure _ 

Ratings  -  Self _ 

Ratings  —  Supervisor _ 


Reenlistment  Rate 
Other _ 


Part  H 


COST-EFFECTIVENESS  ANALYSIS  GUIDANCE 


In  this  part  of  the  specification,  the  approach  to  be  taken  in 
analyzing  the  cost-effectiveness  of  computer-based  training  systems  is 
described.  The  assumptions  which  underly  this  approach  are  discussed 
first  and  general  guidance  is  presented  for  interpreting  and  applying 
the  findings  of  this  type  of  analysis. 

When  examining  alternative  systems,  a  reasonable  cost-effectiveness 
analysis  would  compare  the  best  estimates  of  future  operation  and  main¬ 
tenance  costs  and  not  include  costs  (for  development,  etc.)  that  were 
expended  in  the  past.  In  the  ideal  case,  one  compares  the  cost- 
effectiveness  of  two  alternative  training  systems  when  they  are  in  their 
Operation  and  Maintenance  Phase.  Although  comparisons  are  made  between 
operational  systems  and  alternatives  that  are  still  in  their  Development 
Phase,  these  comparisons  can  only  be  treated  as  hypothetical  or 
"projected."  The  effectiveness  data  that  are  gathered  while  the  system 
is  under  development  can  be  influenced  by  many  factors  which  would  not 
occur  in  a  more  stable  operational  environment.  This  is  particularly 
inportant  to  consider  when  a  prototype  hardware/sof tware  system  is  under¬ 
going  modification  throughout  the  Development  Phase.  Thus,  projected 
cost-effectiveness  ratios  that  are  calculated  during  the  Development  Phase 
should  be  treated  with  caution  as  they  are  subject  to  considerable  error. 


•  An  evaluation  of  cost-effectiveness  can  only  be  made  following 
the  assignment  of  value  or  priority  judgments  to  the  various  dimensions 
of  training  effectiveness.  These  are  to  be  established  by  the  appropriate 
decision-maker  and  should  reflect  training  requirements  and  goals. 


•  The  specific  criteria  by  which  to  compare  the  cost-effectiveness 
of  alternate  training  systems  need  to  be  clearly  established.  If  the 
efficiency  of  a  computer-based  system  in  aiding  the  development  and 
revision  of  training  materials  is  of  prime  importance,  this  aspect  of  the 
system  can  be  assessed  independently  of  its  training  effectiveness. 
Similarly,  a  system's  efficiency  in  scheduling  training  resources  and 
equipment,  keeping  records,  etc.,  can  be  judged.  However,  even  though 
these  measures  are  important  in  determining  the  value  of  a  computer-based 
system,  they  should  not  be  considered  the  same  as  analyzing  a  system's 
cost-effectiveness  when  training  effectiveness  is  of  utmost  concern. 


• - - - - —  - . 
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•  In  the  absence  of  effectiveness  criteria  based  upon  job  or 
training  requirements 3  Indicators  such  as  throughput  should  not  be 
mistaken  for  evidence  of  training  system  quality.  Rather,  in  these  cases, 
such  ratios  (e.g.,  cost  per  student  course  hour)  should  be  interpreted 
as  what  they  are — efficiency  measures. 


•  When  comparing  alternative  training  systems,  identify  a  desired 
level  of  effectiveness  and  examine  the  costs  of  achieving  that  level. 

The  cost-effective  decision  is  to  select  the  alternative  with  the  lowest 
cost. 


or 


Designate  the  acceptable  costs  and  determine  the  level  of  effec¬ 
tiveness  that  different  training  systems  might  reach  assuming  those 
expenditures.  The  cost-effective  decision  is  to  select  the  alternative 
which  yields  the  best  effects. 


•  If  both  costs  and  effectiveness  levels  are  allowed  to  vary,  one 
may  fallaciously  attempt  to  minimize  cost  and  maximize  gain  at  the  same 
time.  Thus,  infinite  effectiveness  at  zero  cost  becomes  an  ideal  which 
is  not  tenable  and  should  not  guide  the  cost-effectiveness  analysis. 


•  The  problems  associated  with  comparing  alternative  training 
systems  which  differ  in  costs  and  effectiveness  are  numerous.  This  is 
of  particular  importance  when  there  are  significant  unanticipated  benefits 
elicited  by  the  more  costly  system  (e.g.,  new  skills  are  taught  which  can 
transfer  to  other  training  situations).  Additional  study  is  then  needed 
to  determine  the  value  of  such  benefits.  The  higher  cost  alternative 
should  be  rejected  if  additional  benefits  do  not  justify  the  extra  expense. 


•  In  performing  a  cost-effectiveness  analysis  of  a  computer-based 
training  system,  one  should  consider  all  appropriate  costs  and  benefits 
during  the  entire  operational  life  of  the  system.  In  this  way,  "cost 
avoidance"  factors  (e.g.,  fewer  instructional  personnel  or  facilities, 
less  training  time,  etc.)  which  may  accrue  over  several  years  will  be 
permitted  to  surface  and  balance  the  large  initial  capital  investment 
costs  of  implementing  a  computer-based  training  system. 


•  Although  this  specification  focuses  on  quantifiable  cost- 
effectiveness  data,  a  thorough  analysis  should  consider  these  data  in 
light  of  any  effects  (both  positive  and  negative)  which  can  only  be 
described  in  qualitative  terms. 
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•  If  training  system  alternatives  are  to  be  compared,  such  a 
comparison  is  meaningful  only  if  the  systems  contain  courses  with 
similar  objectives,  content,  testing  conditions  and  criteria. 


•  If  one  wants  to  compare  media  within  a  course,  cost-effectiveness 
estimates  will  be  error-prone  unless  content,  objectives,  testing 
conditions,  and  criteria  are  equivalent. 


•  Do  not  use  the  cost-effectiveness  ratios  or  relationships 
established  in  the  analyses  as  evidence  of  causality. 


•  In  order  to  attribute  the  effectiveness  of  a  computer-based 
training  system  to  its  innovative  instructional  technology,  one  must  tie 
the  effectiveness  measures  (e.g. ,  criterion  achievement  or  time  scores) 
to  that  portion  of  the  course  that  is  supported  by  this  technology.  If 
the  portion  of  a  course  that  is  uniquely  related  to  the  innovation  or 
method  being  evaluated  cannot  be  identified,  then  by  definition  no 
conclusions  can  be  drawn  related  to  the  efficacy  or  cost-effectiveness 
of  that  innovation  or  method. 
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INSTRUCTIONS  FOR  CALCULATING 
PROJECTED  COST-EFFECTIVENESS  RATIOS 


In  this  part  of  the  specification,  instructions  and  worksheets  are  provided  for 
calculating  Projected  cost-effectiveness  ratios  which  can  be  used  to  evaluate  a  computer- 
based  training  system  in  its  Development  Phase  in  comparison  with  alternative  sys- 
tem(s).  There  are  many  alternative  cost-effectiveness  ratio  which  might  be  considered 
appropriate  for  particular  applications.  It  is  possible  that  several  ratios  will  be  required 
in  order  to  make  a  decision  concerning  whether  one  system  is  better  than  another.  This 
will  require  the  decision  maker  to  rate  the  relative  importance  of  one  measure  of  effec¬ 
tiveness  over  another  and  make  comparisons  between  systems  by  calculating  only  those 
ratios  using  the  most  important  measures. 

The  first  two  ratios  are  indices  of  efficiency  and  are  useful  as  measures  of  a  training 
system’s  cost-effectiveness-PROVIDED  that  training  requirements  have  been  established 
and  graduation  signifies  attainment  of  these  criteria.  Following  these  ratios  is  a  worksheet 
upon  which  to  rate  the  importance  of  effectiveness  measures. 


0  PROJECTED  GRADUATION  COST 

This  index  provides  the  Projected  ratio  of  total  dollars  expended  to  produce  a  graduate 
for  all  Courses  supported  by  the  system.  If  you  are  going  to  calculate  a  ratio  for  each 
Course  separately,  indicate  this  Course  on  the  worksheet. 


Total  #  _  Total  # 

Graduates  Students' 


r~Total  #  Estimated 

I  Students'  *  Attrition  Rate2 

'Total  # Students  From  Part  A,  p.  3. 

2 Attrition  Rate  From  Part  G,  p.  89. 


Projected  Graduation  Cost 


Total  Estimated*  Operation  &  Maintenance  Costs  (p.  66,  Vol.  Ill) 
Total  Number  of  Graduates  (all  Courses) 


This  ratio  can  be  calculated  by  course,  but  there  would  need  to  be  an  appropriate  allocation 
of  Estimated  Operation  &  Maintenance  Costs  to  each  Course  supported  by  the  system. 


•In  the  Development  Phase,  Operation  &  Maintenance  costs  can  only  be  Estimated.  You  may  use 
Volume  III  of  this  specification  to  estimate  Operation  &  Maintenance  costs  for  use  in  these  calculations. 
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WORKSHEET  FOR  COST-EFFECTIVENESS  CALCULATIONS 


PROJECTED  GRADUATION  COST 


Total  Estimated"  Operation  _ 

and  Maintenance  Costs - 

(From  page  66,  Vol.  Ill) 


Total  # 
Graduates 


Total  Students 
(From  page  3) 


Total  Students 
(From  page  31 


Check  Course(s) 
Involved  in  Analysis 

□  Course  _ 

□  Course  _ 

□  Course _ 

□  All  Courses 


X 


Estimated 
Attrition  Rate 
(From  page  89) 


(Total  #  Graduates) 


(Estimated  O  &  M  Costs) 

Projected  Graduation  Cost  =  $ _ 


$ 


/graduate 


(Total  #  Graduates) 


PROJECTED  GRADUATION  COST 


Total  Estimated *  Operation 
and  Maintenance  Costs 


$  _ 

(From  page  66,  Vol.  III! 


Total  # 
Graduates 


Total  Students 
(From  page  3) 


Total  Students 
(From  page  3) 


Check  Course(s) 
Involved  in  Analysis 

□  Course  _ 

□  Course  _ 

□  Course  _ 

□  All  Courses 


X 


Estimated 
Attrition  Rate 
(From  page  89) 


(Total  #  Graduates) 


(Estimated  O  &  M  Costs) 

Projected  Graduation  Cost  =  $ _ 


$ 


/graduate 


(Total  #  Graduates) 
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(D  PROJECTED  HOURLY  COST  OF  INSTRUCTION 

This  index  provides  the  Projected  ratio  of  dollars  expended  for  each  student  hour 
in  the  Courses  supported  by  the  system.  If  you  are  going  to  calculate  this  ratio  for 
each  Course  or  Course  Section  separately ,  indicate  the  specific  Course  and/or  Section 
on  the  work  sheet. 


To  calculate  the  instruction  hours  for  each  Course  use  the  following  formula: 


Total  Average  #  Students/ 

Course  -  Course  x  Course/Year2 

Hours  Time1 


1  Average  Course  Time  From  Pari  G.  p.  87;  or 
Part  F  Summary,  p.  83. 

2#  Students/Course/Year  From  Part  A,  p.  3. 


Total  Instructional  Hours  =  Total  Course  Hours  added  together  for  all 

Courses  supported  by  the  system. 


Projected  Hourly  _  Total  Estimated*  Operation  &  Maintenance  Costs  (p.  66,  Vol.  Ill) 
Cost  of  Instruction  Total  Instructional  Hours  (all  Courses) 


If  this  ratio  is  calculated  by  Course,  use  Total  Course  Hours  in  the  denominator. 
However,  Estimated  Operation  &  Maintenance  Costs  then  would  need  to  be  allocated 
to  each  Course  supported  by  the  system.  This  Projected  ratio  can  also  be  calculated 
by  Course  Section.  Here  too.  Estimated  Operation  &  Maintenance  Costs  would  need 
to  be  appropriately  allocated  by  Section.  If  this  ratio  is  calculated  by  mode/medium 
of  instruction  (e.g.,  CA1  portions  only),  then  there  is  a  need  to  allocate  Estimated 
Operation  &  Maintenance  Costs  by  type  of  instruction.  This  approach  is  practical 
only  in  Courses  with  Sections  using  a  single  mode/medium  of  instruction. 


•In  the  Development  Phase,  Operation  &  Maintenance  costs  can  only  be  Estimated .  Use  Volume 
III  of  this  specification  to  estimate  the  Operation  &  Maintenance  costs  for  these  calculations. 
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©  PROJECTED  HOURLY  COST  OF  INSTRUCTION 


•  Course 


Section 


Total  Course  Hours  = _ X  _ 

Average  Courts  Time  Students/Year 


•  Course 


Section 


Total  Course  Hours  =  _  X  _ 

Averate  Course  Time  Studentt/Year 


•  Course 


Section 


Total  Course  Hours  =  _  X  _ 

Average  Course  Time  Students/Year 

Total  Instructional  Hours  (All  Courses)  =  _ hours 


Total  Estimated  Operation  &  Maintenance  Costs  (All  Courses)  -  $ 


(From  page  66.  Vol.  Ill) 


Projected  Hourly  Cost  of  Instruction  =  $ 


(Estimated  O  &  M  Costs) 


(Total  Instructional  Hours) 


©  PROJECTED  HOURLY  COST  OF  INSTRUCTION 


•  Course 


Section 


Total  Course  Hours 


Average  Course  Time  Students/Year 


•  Course 


Section 


Total  Course  Hours  =  _  X  _ 

Average  Course  Time  Students/Year 


•  Course 


Section. 


Total  Course  Hours  =  _  X  _ 

Average  Course  Time  Students/Year 

Total  Instructional  Hours  (All  Courses)  =  _ hours 


Total  Estimated  Operation  &  Maintenance  Costs  (All  Courses)  =  $ 


(From  page  66,  Vol.  Ill) 


(Estimated  O  &  M  Costs) 


Projected  Hourly  Cost  of  Instruction  =  $ . 


(Total  Instructional  Hours) 
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®  COMPARISON  OF  EFFECTIVENESS  MEASURES 

In  order  to  calculate  the  Projected  cost-effectiveness  ratios  that  are  most  important 
to  your  analysis,  the  effectiveness  measures  specified  in  Part  G*  need  to  be  rated  in 
importance  by  the  decision-maker. 


(a)  List  the  end-of-course  effectiveness  scores  to  be  used  in 
decision-making  (from  Part  G). 

®  Estimate  the  "value”  of  each  score  by  rating  the  importance 
of  each  measure  based  on  the  system’s  or  Course’s  training 
requirements.  (Place  a  rating  of  1  to  10  in  the  space 
provided— 10  being  most  important.) 


Projected  cost-effectiveness  ratios  can  then  be  calculated  for  each  of  the  “most  important” 
measures  before  considering  some  of  the  other  effectiveness  scores. 


For  example,  if  the  Achievement  level  attained  as  reflected  by  Accuracy  Scores  is 
the  most  important  measure  of  training  effectiveness,  then  it  should  be  the  basis 
for  comparing  alternative  training  systems.  Projected  cost-effectiveness  comparisons 
can  be  made  using  this  indicator  (assuming  costs  are  held  constant);  or  if  achievement 
scores  are  held  constant,  then  training  systems  can  be  compared  with  regard  to  the 
costs  required  to  reach  this  level  of  achievement. 


•Pert  F  Summary,  if  Part  G  not  available. 
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WORKSHEET  FOR  COST-EFFECTIVENESS  CALCULATIONS 


(5)  Ratings  of  Effectiveness  Measures 


Time  Measures 

Average  Course  Time 

Achievement  Measures 

Accuracy /Speed  Scores 
Number  Objectives  Passed 
Percent  Students  Passed 
Gain  Scores 

#  Attempts  to  Pass  Criterion  Test 

Other  Measures 

Absentee  Rates 
Attrition  Rates 
Recycle  Rates 
Instructor  Ratings 
Attitude  Scores 
Other 

Transfer  Measures 

Subsequent  Course  Time 
Subsequent  Course  Achievement 
Job  Performance  -  Time  to  Qualification 
Job  Test  Scores 
Work  Productivity 
Self-Ratings 
Supervisory  Ratings 
Absentee  Rates 
Attitudes 

Reenlistment  Rate 
Other 


Course 


®  ® 


End-of -Course 
Effectiveness  Scores 
(From  Part  G, 
pp.  87, 89  &  91)* 


Ratings  of 
Importance 
<1-10. 10  « 
Most  Important) 


•Part  F  Summary,  if  Part  G  not  available. 
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Make  at  many  copies  as  required 
for  the  number  of  Courses. 


1 


